Service Call Liveness
Overview
A service proxy call travels over the OS bus to whichever node hosts the service — another server's node for a Java service, or a gateway socket for a TS service running on a workload node. If that node dies while the call is in flight, nothing would ever answer the caller. Kinotic handles this as a property of the platform rather than of each call: no service call carries a timeout, and none needs one. Instead, every hop that holds a pending request watches the registration of the callee that serves it, and fails the request the moment that registration is gone.
The reason to prefer this over a timeout is that a timeout encodes a guess about how long work takes, and no number is right for a service the platform does not own. Liveness observes whether the worker still exists. A legitimate ten-hour call is never cut short; a dead node fails its pending calls within seconds.
Registration is liveness
A service is reachable exactly while its address has an entry in the cluster's replicated subscription cache. That cache is maintained by the Vert.x cluster manager on Ignite, and Ignite removes a dead node's entries as part of its own failure detection. The platform already turns those updates into a push signal: the callee side of a streaming call watches the caller's reply destination and cancels the stream when the caller disappears; for a TypeScript service the gateway holding its connection does the watching and delivers the cancel over the socket. Liveness for callers is the same signal, in the other direction.
Reading the diagram: every pending request is a lease pinned to the node whose acknowledgement took it, held at whichever hop has cluster visibility. A Java caller and the MCP invoker are in the cluster and take leases directly through the request liveness watcher. A TS client — the UI, or a TS service calling another service — is outside the cluster and cannot see registrations, so the gateway takes the leases on its behalf and answers the client with an error reply when one fails. The watcher holds one cluster membership subscription per node, however many requests are in flight, and it rides on the discovery events the cluster already delivers to every node; nothing new is replicated. The count of requests it holds is the rpc.pending.requests gauge.
The signal that ends a request is the same one every reply already carries: a terminal reply is marked control: complete on the event itself, so any hop can release its lease on that one event without knowing whether the method returned a value, nothing, or a stream.
A call in flight when its node dies
The ack that confirms receipt of a request names the node whose consumer took it. That matters for any address with more than one registration: a service published on several nodes, several TS instances of one service, or the shared address a scoped service joins for its @ScopeOptional methods. A call to such an address lands on one instance, the address stays active as long as any instance lives, and the caller must know which one took its request.
A lease starts at the ack. Before it, Vert.x itself bounds the send: the request fails at once when nothing is registered, and after its delivery timeout when the ack never arrives — a bound on one network hop, not on the call, since the consumer acks before it dispatches. From the ack on, the lease is pinned to the node that answered and fails when that node leaves the cluster, even while other instances keep the address active, and even after that node has unregistered the address to drain before shutting down. A node that finishes a call before it leaves never fails it.
For a TS service the ack names the gateway holding the instance's subscription, which cannot tell two TS instances on the same gateway apart. The gateway can: it forwarded the invocation to one specific socket and sees the terminal reply come back through the same connection, so it tracks each outstanding invocation per socket and synthesizes the failure for all of them when that socket closes. STOMP heartbeats make a socket close deterministic even when a VM vanishes without closing it: the gateway offers and expects a 30 s heartbeat in both directions and closes a connection silent for two intervals, so this path is bounded by the heartbeat interval — ahead of Ignite's own detection window. The TS client offers and expects the same 30 s, so a gateway that vanishes without closing the socket is a connection loss on the client within two intervals as well, and the calls on it fail through the rule below.
What the caller receives, in every runtime:
| Exception | Meaning | The call… |
|---|---|---|
RpcMissingServiceException | rejected at send, no registration for the address | never executed |
RpcServiceUnavailableException | the node that took the call left the cluster while it was in flight, its acknowledgement never arrived, or the service stopped while producing a stream | may or may not have executed |
Callers of non-idempotent operations should treat the second as ambiguous and check before retrying. In TypeScript a failed call rejects with an RpcError, whose exceptionName and exceptionClass name the exception the server caught — for a service hosted in TypeScript, the class of the error it threw, whose name fills both — so a caller branches on exceptionName === 'RpcServiceUnavailableException' rather than on the message.
A service that stops, on a rolling update or any other graceful shutdown, stops accepting calls first, then answers every call it already has before its node leaves the cluster, so those calls complete normally and no lease fails. Streams it was producing cannot be finished; each one ends with RpcServiceUnavailableException at its subscriber the moment the service stops. The wait for the in-flight calls is bounded, so a call that never finishes cannot hold the shutdown.
Reconnects and node rollovers
A call is bound to the connection it was made on. When that connection closes, for any reason, both ends let go at the same moment: the gateway unregisters the connection's reply consumer and releases its leases, so a server stream producing for it is cancelled through the reply-listener path and no reply is held anywhere, and the client fails every call it had in flight with Connection lost, streams included, the moment it observes the close. A reconnect starts clean. Nothing waits for a client to come back, so nothing can wait on the wrong node, and a node that dies takes no client's pending replies with it because it was holding none.
This is the deliberate choice over holding replies for a returning client. The STOMP port sits behind a load balancer with no session affinity, so a reconnect lands on any gateway node; buffering replies on the node that lost the connection would only pay off with a cross-node handoff, and a handoff still loses the buffer when that node dies. Retrying a call is the cheaper contract. For a caller that needs a message to survive its connection, that is a delivery guarantee, and it belongs in a queue behind the API with acknowledgement and redelivery, not in a reply buffer on a gateway.
What this does not cover
A callee that is alive and registered but wedged — deadlocked, or looping — produces no liveness signal, and no mechanism short of a deadline could catch it. That is a misbehaving service rather than a failed node, and the watcher exposes the count of pending requests per node so that it becomes visible in observability as a count that climbs on a healthy node. If it ever needs bounding, a deadline declared by the service on its own contract composes with everything above; a timeout guessed by the caller does not.