Observability

Overview

Kinotic provides deep observability across your applications, aggregated at multiple levels: System, Organization, and Application.

Metrics

Real-time monitoring of CPU, memory, and data throughput for all running services. Metrics are collected automatically for every deployed application and available through the Kinotic dashboard.

Traces and Spans

Drill from high-level overviews into detailed execution logs. Distributed tracing follows requests across service boundaries, so you can pinpoint performance bottlenecks and errors in complex service interactions. A workload running in a micro VM ships its traces and metrics through the node that runs it — see Workload Traces and Metrics — and the portal shows them on an application's Observability page, and across the organization at Observability in the organization sidebar.

LLM Observability

Trace user interactions with LLMs and track token utilization for cost analysis. LLM request and response data is indexed via Grafana Loki, giving you full-text search across all LLM interactions with filtering by user, application, model, and time range.

Audit Logs

Track platform activity including:

• Login history — Who connected, when, and from where
• Configuration changes — OIDC provider updates, LLM configuration changes, and application settings modifications
• Activity counts — Aggregate usage metrics per user, application, and organization

Application Logs

View microservice logs directly from the dashboard with the ability to temporarily adjust logging levels for debugging. Increase verbosity on a running service to investigate an issue, then restore normal levels when done — no redeployment required.

The same dialog edits a node's trace log filters, so turning a logger up to TRACE does not have to mean drowning in whatever service talks most. Both changes last until the node restarts.

Workload Logs

Logs from micro VM workloads (builds, deploys, and application containers) are shipped to Grafana Loki and can be tailed live or queried historically, per workload.

Log shipping architecture

Every node runs a managed Grafana Alloy process whose pipeline is regenerated as workloads come and go; Alloy tails each running workload's log files and pushes them to Loki, and receives the traces and metrics of the workloads that elect telemetry and pushes them to Tempo and Mimir (see Workload Traces and Metrics). What a workload has to do for its logs to be shipped depends on the node's VM provider:

CLOUD_HYPERVISOR nodes label each stream stdout or stderr. Both providers bound what a workload's logs occupy on the node through logPolicy — maxSizeMb is the size at which the current file rotates, and maxFiles how many rotated files are kept beside it — the container runtime enforcing it on CLOUD_HYPERVISOR, and the image itself on BOXLITE.

Workload VMs run detached from the vm-manager process by default (Workload.detached), and the vm-manager persists each workload's state on the node. If the vm-manager restarts (a crash, a systemd restart), it reattaches to the detached VMs that are still running and regenerates the Alloy pipeline, so their logs keep shipping. A non-detached workload runs in the foreground — the call that starts it resolves only once its run has ended, with the exit code — and ends with the vm-manager process.

On a deployment's job run page the Sync project source step expands into the sync workload's log, the way a CI step does: open while the step runs, tailing live, and readable afterwards from the run's history. A workload whose run has ended keeps its log files on the node until it is destroyed, and a destroy waits for the shipper to read them to the end first, so a run over in seconds still ships every line it wrote.

A stopped workload can be restarted in place (restartWorkload) unless it was stopped with Workload.autoRemove, which discards the VM and its disk at stop. A restart boots the same VM, so its log streams continue under the same vm_id label.

Every log stream carries these labels:

Loki runs multi-tenant. A workload's logs are stored in its organization's tenant (X-Scope-OrgID = the organization id), so one organization's queries can never see another's streams. Platform workloads with no organization ship to the reserved kinotic-system tenant — organization ids beginning with kinotic are reserved for the platform. Platform operators can query across tenants with pipe-separated ids (for example acme|kinotic-system).

Reading workload logs

The LogService (@kinotic-ai/management-api) streams (tail) and queries (history) the logs of workloads the caller may view: an organization participant sees its own organization's workloads, a system participant sees any. Both methods return raw Loki response bytes for the caller to parse.

Workload Traces and Metrics

A workload that sets telemetry to true is given an OTLP endpoint of its own on the node it runs on, and the traces and metrics it exports there are shipped to Tempo and Mimir under its organization's tenant, carrying the same identity as its log streams. Telemetry is elected per workload: the node ships nothing for a workload that leaves it off, whatever the workload's runtime exports. The runtime workload of every deployed project elects it, under the project's name as its service name.

What the node does

When the workload's VM is created the node issues it an endpoint — a host port its Alloy listens on for that workload alone, and a bearer token only that guest is given — and names it in the guest environment through the standard OpenTelemetry variables, which every OpenTelemetry SDK reads:

The exporter variables are laid over the workload's own environment and secrets, so a workload cannot point its exporter elsewhere by setting them itself; the service name is the one variable the workload may set, since it is how its spans and metrics are grouped. The endpoint holds for the workload's life, restarts included, and is released when the workload is destroyed. Each receiver accepts only its own token, so a guest cannot push into another workload's stream. A workload electing telemetry on a node that ships neither signal runs without an endpoint.

What the workload must do

Export over OTLP from that environment. An OpenTelemetry SDK configured from the environment — the Java agent, or the Node SDK with its default configuration — does so without any code of its own. The Kinotic runtimes instrument every service invocation through the OpenTelemetry API, so a Kinotic application whose process runs such an SDK ships a span per call, continuing the trace of the caller that invoked it. A runtime that ignores the variables ships nothing.

A workload with network.mode DISABLED has no way to reach the endpoint, so a node refuses one that also elects telemetry rather than starting it silent.

How a guest reaches its endpoint

The port is opened by the same per-workload rules as the workload's egress, so a CLOUD_HYPERVISOR node that does not deny egress by default refuses a workload electing telemetry, as it refuses one declaring allowedHosts.

Identity and tenant

Every span and every metric arrives with these resource attributes, matching the labels on the workload's log streams:

Traces and metrics are pushed with X-Scope-OrgID set to the organization id, or kinotic-system for platform workloads with no organization — the same tenant the logs go to. Tempo and Mimir run multi-tenant (multitenancy_enabled: true, as the docker-compose stack sets them), so each organization's telemetry is kept apart the way Loki keeps its logs, and Tempo's metrics-generator writes the span metrics it derives — traces_spanmetrics_calls_total, traces_spanmetrics_latency_bucket — into the same tenant, carrying application_id and workload_id beside service so one application's RED metrics can be selected within its organization.

Reading workload traces and metrics

The TelemetryService (@kinotic-ai/management-api, Kinotic.telemetry) searches traces (searchTraces, a TraceQL query over a time range), fetches one trace with all its spans (findTrace), and evaluates PromQL (queryMetrics, a range query at a step). Each call names an organization: an organization participant may name only its own, and a system participant any, or none for the platform's own telemetry. The tenant is the boundary; what a query selects within it is the caller's to decide. All three return the raw Tempo and Prometheus response bytes for the caller to parse.

The portal shows them on an application's Observability page: a trace search — by service, span name, errors, and duration — opening each trace on its own page as a waterfall of its spans with their attributes, and the requests, errors, and latency of the application's services beside a free PromQL query, the errors chart leading to the traces with a failed span. The organization's Observability page has the same view across all of its applications or one of them, and so does the system console at Organizations → (organization) → Observability, with each application's view under Applications → (application) → Observability. The console's own Observability page, under Platform, reads the system tenant: the traces and metrics of the servers, the gateway, and the worker nodes themselves.

Configuration

The vm-manager resolves the Alloy binary from the PATH, downloading its pinned release when none is found. Both the download and Alloy's launch happen while the node starts up, before it registers and accepts workloads, so no workload operation waits on them. One Alloy process ships logs, traces, and metrics; a node configured for some of them runs it for those alone.