Observability

Overview

Kinotic provides deep observability across your applications, aggregated at multiple levels: System, Organization, and Application.

Metrics

Real-time monitoring of CPU, memory, and data throughput for all running services. Metrics are collected automatically for every deployed application and available through the Kinotic dashboard.

Traces and Spans

Drill from high-level overviews into detailed execution logs. Distributed tracing follows requests across service boundaries, so you can pinpoint performance bottlenecks and errors in complex service interactions. A workload running in a micro VM ships its traces and metrics through the node that runs it — see Workload Traces and Metrics — and the portal shows them on an application's Observability page, and across the organization at Observability in the organization sidebar.

LLM Observability

Trace user interactions with LLMs and track token utilization for cost analysis. LLM request and response data is indexed via Grafana Loki, giving you full-text search across all LLM interactions with filtering by user, application, model, and time range.

Audit Logs

Track platform activity including:

• Login history — Who connected, when, and from where
• Configuration changes — OIDC provider updates, LLM configuration changes, and application settings modifications
• Activity counts — Aggregate usage metrics per user, application, and organization

Application Logs

View microservice logs directly from the dashboard with the ability to temporarily adjust logging levels for debugging. Increase verbosity on a running service to investigate an issue, then restore normal levels when done — no redeployment required.

The same dialog edits a node's trace log filters, so turning a logger up to TRACE does not have to mean drowning in whatever service talks most. Both changes last until the node restarts.

The servers' own logs are in Loki's kinotic-system tenant, and every stream of them carries two labels: service_name, the server (kinotic-server-management, kinotic-server-system or kinotic-server-app), and service_instance_id, the node. Each node reads both from its OTEL_SERVICE_NAME and the service.instance.id in its OTEL_RESOURCE_ATTRIBUTES, and every deployment sets them to match how its logs are shipped:

The system console's Cluster page lists each server node with the server it runs (management, system or app) and its Kinotic version, and its Logs button shows that node's logs alone. A node with no service.instance.id shows every node of its server interleaved, and one with no OTEL_SERVICE_NAME shows no button. LogService.tailServer and LogService.serverHistory read them, for a system participant alone.

The log view, here and for a workload's logs, loads the selected time range and follows new lines only once its Follow button is pressed. It shows an entry that spans several lines as one row per line under the entry's timestamp, and renders the ANSI color and style escapes a line carries as a terminal would. An entry shipped with an OpenTelemetry severity_text shows that level beside its timestamp, colored as Spring Boot's console colors it, and an entry shipped with an exception_stacktrace (the OpenTelemetry agent keeps a logged exception there rather than in the message) shows the stack trace under its message.

Workload Logs

Logs from micro VM workloads (builds, deploys, and application containers) are shipped to Grafana Loki and can be tailed live or queried historically, per workload.

The workload runner starts the processes it runs (the sync's install, entity sync and UI builds, and an application's microservice) with FORCE_COLOR=1, unless the workload sets its own FORCE_COLOR, so tools that honor it, such as bun, chalk and picocolors, keep the ANSI colors they would print to a terminal, and the log view renders them.

Log shipping architecture

Every node runs a managed Grafana Alloy process whose pipeline is regenerated as workloads come and go; Alloy tails each running workload's log files and pushes them to Loki, and receives the traces and metrics of the workloads that elect telemetry and pushes them to Tempo and Mimir (see Workload Traces and Metrics). What a workload has to do for its logs to be shipped depends on the node's VM provider:

CLOUD_HYPERVISOR nodes label each stream stdout or stderr. Both providers bound what a workload's logs occupy on the node through logPolicy — maxSizeMb is the size at which the current file rotates, and maxFiles how many rotated files are kept beside it — the container runtime enforcing it on CLOUD_HYPERVISOR, and the image itself on BOXLITE.

Workload VMs run detached from the vm-manager process by default (Workload.detached), and the vm-manager persists each workload's state on the node. If the vm-manager restarts (a crash, a systemd restart), it reattaches to the detached VMs that are still running and regenerates the Alloy pipeline, so their logs keep shipping. A non-detached workload runs in the foreground — the call that starts it resolves only once its run has ended, with the exit code — and ends with the vm-manager process.

On a deployment's job run page the Sync project source step expands into the sync workload's log, the way a CI step does: open while the step runs, where its Follow button follows new lines as they arrive, and readable afterwards from the run's history. The node's removal of an ended run waits for the shipper to read the workload's log files to the end first, so a run over in seconds still ships every line it wrote, and a workload's logs outlive its VM: they are read by the organization and workload id, so the sync and publish workloads, which the node removes as their runs end, stay readable from the run's history until the workload's record is deleted — from the console, or by the retention sweep once the run is older than kinotic.systemApi.workload.retentionDays — which deletes its log streams with it (see Workload retention).

A run that has ended — stopped, exited, or failed — is removed from the node once the server holds its outcome: the node ships what the VM wrote, then removes the VM, its disk and its log files. The workload's record and its logs remain until the record is deleted, so a microservice restart from the console runs a fresh VM with a new vm_id.

Every log stream carries these labels:

Loki runs multi-tenant. A workload's logs are stored in its organization's tenant (X-Scope-OrgID = the organization id), so one organization's queries can never see another's streams. Platform workloads with no organization ship to the reserved kinotic-system tenant — organization ids beginning with kinotic are reserved for the platform. Platform operators can query across tenants with pipe-separated ids (for example acme|kinotic-system).

Reading workload logs

The LogService (@kinotic-ai/management-api) streams (tail) and queries (history) the logs of workloads the caller may view: an organization participant sees its own organization's workloads, a system participant sees any. Its tailServer and serverHistory read a platform server's logs the same way (see Application Logs). A history covers its range up to but excluding its end, and a tail starts at the moment it is given, including it: it first sends the entries since then and then each new one, so a tail started at a history's end reads on from it with no entry repeated or missed, which is how the console's log view follows. Every method returns raw Loki response bytes for the caller to parse.

Workload Traces and Metrics

A workload that sets telemetry to true is given an OTLP endpoint of its own on the node it runs on, and the traces and metrics it exports there are shipped to Tempo and Mimir under its organization's tenant, carrying the same identity as its log streams. Telemetry is elected per workload: the node ships nothing for a workload that leaves it off, whatever the workload's runtime exports. The runtime workload of every deployed project elects it, under the project's name as its service name.

What the node does

When the workload's VM is created the node issues it an endpoint — a host port its Alloy listens on for that workload alone, and a bearer token only that guest is given — and names it in the guest environment through the standard OpenTelemetry variables, which every OpenTelemetry SDK reads:

The exporter variables are laid over the workload's own environment and secrets, so a workload cannot point its exporter elsewhere by setting them itself; the service name is the one variable the workload may set, since it is how its spans and metrics are grouped. The endpoint holds for the workload's life, restarts included, and is released when the workload is destroyed. Each receiver accepts only its own token, so a guest cannot push into another workload's stream. A workload electing telemetry on a node that ships neither signal runs without an endpoint.

What the workload must do

Export over OTLP from that environment. An OpenTelemetry SDK configured from the environment — the Java agent, or the Node SDK with its default configuration — does so without any code of its own. The Kinotic runtimes instrument every service invocation through the OpenTelemetry API, so a Kinotic application whose process runs such an SDK ships a span per call, continuing the trace of the caller that invoked it. A runtime that ignores the variables ships nothing.

A deployed project needs nothing of its own: the workload-runner image preloads the Node SDK into the microservice process, started only when the endpoint variable is present and flushing its pending spans when the process is stopped. The SDK registers against the global OpenTelemetry API, which the project's own @opentelemetry/api — the one @kinotic-ai/core records through — resolves as long as it is the same major version and no newer than the image's. A project that starts an SDK of its own finds the API already registered, so that SDK exports nothing: leave the bootstrap to the image.

A workload with network.mode DISABLED has no way to reach the endpoint, so a node refuses one that also elects telemetry rather than starting it silent.

How a guest reaches its endpoint

The port is opened by the same per-workload rules as the workload's egress, so a CLOUD_HYPERVISOR node that does not deny egress by default refuses a workload electing telemetry, as it refuses one declaring allowedHosts.

Identity and tenant

Every span and every metric arrives with these resource attributes, matching the labels on the workload's log streams:

Traces and metrics are pushed with X-Scope-OrgID set to the organization id, or kinotic-system for platform workloads with no organization — the same tenant the logs go to. Tempo and Mimir run multi-tenant (multitenancy_enabled: true, as the docker-compose stack and the Kubernetes deployment set them), so each organization's telemetry is kept apart the way Loki keeps its logs, and Tempo's metrics-generator writes the span metrics it derives — traces_spanmetrics_calls_total, traces_spanmetrics_latency_bucket — into the same tenant, carrying application_id and workload_id beside service so one application's RED metrics can be selected within its organization.

The platform servers name the backend each of their HTTP client spans called as its peer.service — elasticsearch, loki, tempo, or mimir — and the span metrics carry it as peer_service, so a query over the system tenant can select those calls or leave them out: sum by (span_name) (rate(traces_spanmetrics_calls_total{peer_service!="elasticsearch"}[5m])) charts the servers' work without the bare GET / POST spans of their Elasticsearch calls.

Reading workload traces and metrics

The TelemetryService (@kinotic-ai/management-api, Kinotic.telemetry) searches traces (searchTraces, a TraceQL query over a time range), fetches one trace with all its spans (findTrace), and evaluates PromQL (queryMetrics, a range query at a step). Each call names an organization: an organization participant may name only its own, and a system participant any, or none for the platform's own telemetry. The tenant is the boundary; what a query selects within it is the caller's to decide. All three return the raw Tempo and Prometheus response bytes for the caller to parse.

The portal shows them on an application's Observability page: a trace search — by service, span name, errors, and duration — opening each trace on its own page as a waterfall of its spans with their attributes, and the requests, errors, and latency of the application's services beside a free PromQL query, the errors chart leading to the traces with a failed span. The organization's Observability page has the same view across all of its applications or one of them, and so does the system console at Organizations → (organization) → Observability, with each application's view under Applications → (application) → Observability. The console's own Observability page, under Platform, reads the system tenant: the traces and metrics of the servers, the gateway, and the worker nodes themselves.

Configuration

The vm-manager resolves the Alloy binary from the PATH, downloading its pinned release when none is found. Both the download and Alloy's launch happen while the node starts up, before it registers and accepts workloads, so no workload operation waits on them. One Alloy process ships logs, traces, and metrics; a node configured for some of them runs it for those alone.