Observability

Overview

Kinotic provides deep observability across your applications, aggregated at multiple levels: System, Organization, and Application.

Metrics

Real-time monitoring of CPU, memory, and data throughput for all running services. Metrics are collected automatically for every deployed application and available through the Kinotic dashboard.

Traces and Spans

Drill from high-level overviews into detailed execution logs. Distributed tracing follows requests across service boundaries, so you can pinpoint performance bottlenecks and errors in complex service interactions.

LLM Observability

Trace user interactions with LLMs and track token utilization for cost analysis. LLM request and response data is indexed via Grafana Loki, giving you full-text search across all LLM interactions with filtering by user, application, model, and time range.

Audit Logs

Track platform activity including:

• Login history — Who connected, when, and from where
• Configuration changes — OIDC provider updates, LLM configuration changes, and application settings modifications
• Activity counts — Aggregate usage metrics per user, application, and organization

Application Logs

View microservice logs directly from the dashboard with the ability to temporarily adjust logging levels for debugging. Increase verbosity on a running service to investigate an issue, then restore normal levels when done — no redeployment required.

Workload Logs

Logs from micro VM workloads (builds, deploys, and application containers) are shipped to Grafana Loki and can be tailed live or queried historically, per workload.

Log shipping architecture

Each vm-manager node mounts a per-workload host directory into the VM at /var/log/kinotic. Any *.log file the workload writes there is shipped — this is the logging contract for workload images. The node runs a managed Grafana Alloy process whose pipeline is regenerated as workloads come and go; Alloy tails the log directories and pushes to Loki.

Workload VMs run detached from the vm-manager process by default (Workload.detached), and the vm-manager persists each workload's state on the node. If the vm-manager restarts (a crash, a systemd restart), it reattaches to the detached VMs that are still running and regenerates the Alloy pipeline, so their logs keep shipping. A non-detached workload ends with the vm-manager process.

A stopped workload can be restarted in place (restartWorkload) unless it was stopped with Workload.autoRemove, which discards the VM and its disk at stop. A restart boots the same VM, so its log streams continue under the same vm_id label.

Every log stream carries these labels:

Loki runs multi-tenant. A workload's logs are stored in its organization's tenant (X-Scope-OrgID = the organization id), so one organization's queries can never see another's streams. Platform workloads with no organization ship to the reserved kinotic-system tenant — organization ids beginning with kinotic are reserved for the platform. Platform operators can query across tenants with pipe-separated ids (for example acme|kinotic-system).

Reading workload logs

The LogService (@kinotic-ai/os-api) streams (tail) and queries (history) the logs of workloads the caller may view: an organization participant sees its own organization's workloads, a system participant sees any. Both methods return raw Loki response bytes for the caller to parse.

Configuration

The vm-manager resolves the Alloy binary from the PATH, downloading its pinned release on first use when none is found.