Configuration

Overview

Kinotic OS is configured through a combination of application properties, environment variables, and Helm chart values. Key configuration areas include:

• Server configuration — Port bindings, cluster discovery, and runtime settings
• Elasticsearch settings — Connection URLs, index defaults, and replica configuration
• Authentication — OIDC provider configuration, session management, and security policies
• Environment variables — Runtime overrides for containerized deployments
• Helm values — Kubernetes-specific configuration for resource limits, replicas, ingress, and TLS

Loki, Tempo, and Mimir

The server reads workload logs from Grafana Loki, and workload traces and metrics from Grafana Tempo and Mimir (see Observability). Each runs multi-tenant, and the server queries the tenant the caller may see.

In docker-compose the server points at http://loki:3100, http://tempo:3200, and http://mimir:9009; the Helm chart defaults them to http://loki.observability.svc:3100, http://tempo.observability.svc:3100, and http://mimir.observability.svc:9009, the services the Azure terraform deploys in the observability namespace (Tempo's chart serves its HTTP API on 3100).

Every server also runs the OpenTelemetry Java agent its image embeds, which exports the server's own traces and metrics to the platform tenant, kinotic-system, read by the system console's Platform Observability page. The Helm chart attaches it on every server and sends each signal straight to its store, with no collector between; its logs reach Loki through Alloy instead, so the agent exports none:

VM provider

Each vm-manager node runs every workload on one VM provider, chosen by the node rather than by the workloads it is given. The node reports which one it runs when it registers, and the orchestrator surfaces it on the node in the system UI.

CLOUD_HYPERVISOR needs the node's Docker daemon to register a kata-clh runtime and to keep its data root on an XFS filesystem mounted with prjquota. Without the quota support the daemon refuses any workload declaring a diskSizeMb, since that is what caps a container's rootfs. deployment/vm-node in the repository provisions a node that way and installs the vm-manager as a service.

BOXLITE keeps box records and every guest's rootfs disk under one directory. That directory is the filesystem workload disks grow into and the one the node reports its totalDiskMb from, so a node taking workloads of any size wants it on a data volume rather than the root filesystem under the service account's home.

Workload volume mounts

On every node, whichever provider it runs, a workload volume mount must live under the node's workload data directory — mounts are bound with the vm-manager's authority, so this boundary is what keeps a workload spec from mounting an arbitrary host directory. The node creates missing directories for writable mounts, refuses a read-only mount of a directory that does not exist, and reports the directory to the server at registration so deployment flows compose host paths under it. On a CLOUD_HYPERVISOR node the directory should sit on an XFS filesystem with prjquota, since that is what enforces a writable mount's sizeLimitMb; a node whose directory cannot enforce it reports the problem and stops taking workloads.

The default sits under the home directory of the account the vm-manager runs as, so a node creates workload directories with the authority it already has. A node taking real deployments sets it to a path on a data volume — on a CLOUD_HYPERVISOR node, the XFS filesystem whose project quotas enforce sizeLimitMb, which is not where the default lands.

Workload images

A workload's image is pulled before every start when its reference floats — no tag, or latest — so a node runs what the registry holds now rather than the copy it pulled first. A reference pinning a tag other than latest or a digest names one immutable image, and is pulled only when the node does not have it yet. Both providers apply the rule, which is the one Kubernetes uses as its default imagePullPolicy. A BOXLITE node resolves a floating tag to its current digest through the registry itself, anonymously, so its images must be public; a CLOUD_HYPERVISOR node has the container runtime pull the tag.

Workload egress

A CLOUD_HYPERVISOR node denies workload egress by default and permits each workload only what its policy allows. The node's firewall carries the denial; the vm-manager writes one exception per allowed destination when a workload starts, and removes them when it stops.

Every destination comes from the workload's own network.allowedHosts, including the api-gateway — the server sets it there, because only the server knows where the gateway is. The node adds the destinations the workload cannot know: the resolver it was given, and its own OTLP endpoint for a workload that elects telemetry (see Workload Traces and Metrics).

network.allowedHosts entries are IPv4 addresses, CIDRs, or hostnames. An address or CIDR becomes a rule naming it. A hostname is enforced through the node's resolver: every workload is given the node's dnsmasq as its only resolver, and dnsmasq writes each address it answers for an allowed name into an ipset the workload's rule matches — so what the workload reaches is what the name resolved to when it looked it up, however the name's records move, and an address it did not resolve through the node stays denied. A name covers its subdomains: github.com also admits api.github.com. A destination in the same virtual network is best expressed as the subnet's CIDR, which needs no lookup.

A name the node itself pins in /etc/hosts is answered from there for its workloads too, and its pinned address is admitted for that name without expiry; that is how workloads reach the api-gateway by its certificate's name on a LAN, without the traffic leaving through the router.

A node that cannot enforce a hostname — no ipset, or no dnsmasq listening on the resolver its workloads are given — refuses a workload declaring one rather than starting it with the name dropped from its policy.

An empty list means the same on a BOXLITE node, and hostnames are accepted on both providers.

mode: DISABLED leaves no interface to publish a port on or to reach the node's OTLP endpoint through, so a node refuses a workload declaring it together with portMappings or telemetry — on either provider — rather than starting one whose published ports or telemetry cannot work.

A node that does not deny egress by default cannot honour a declared allowlist, so a workload declaring one is refused there rather than started with access it was supposed to be denied.

A node denies every workload the cloud metadata endpoint (169.254.169.254) and the Azure WireServer's control ports (168.63.129.16), which is what keeps a guest from reading the host's own identity. A workload may still be granted one, by naming the address exactly:

An entry that names a protected address is placed above the node's own rules so it overrides them; every other entry is placed below, so a range containing one grants everything in it except that address, and a hostname resolving to one grants nothing there. Only a system participant can reach the service that carries a workload's policy, so naming an address outright is a decision the platform made — a range that merely contains one is not, and does not become one by accident.

A provider needs its host prepared for it, so the setting follows the node's provisioning: a developer machine keeps the default, and a node built for Kata is configured for CLOUD_HYPERVISOR. An unrecognised value stops the vm-manager at startup rather than falling back, so a node never registers running something other than what it was configured for.

Node capacity

Each vm-manager node reports its capacity when it registers with the orchestrator. A workload is placed on the first online node whose unallocated capacity covers the workload's cpus, memoryMb, and diskSizeMb, and that capacity is reserved on the node atomically, so two deployments placed at the same moment can never both take the same room; a deployment whose room was taken in between is placed again on what is left, up to three times.

cpus is in cores and may be a fraction: 0.5 is half a core, enforced as a CPU quota on the VM on CLOUD_HYPERVISOR nodes (a BOXLITE node boots the VM with the whole core the share is part of). The node's freeCpus moves by the same fraction, so a 4-core node holds eight half-core workloads.

The node's record keeps freeCpus, freeMemoryMb and freeDiskMb: the totals less what the workloads running on it are sized for. A run holds everything the workload is sized for from the moment it is placed; when it ends — stopped, exited, failed, or reported gone by the node — everything returns to the node. A run's end is recorded once, so a stop and the node's own report of the same end cannot return the room twice. A node that registers again rebuilds its free capacity from the workload records still running on it, so a run the node ended while the server was away is still accounted for; the write is conditional on the counters it overwrites, so a reservation or a release landing between the records' read and the write is answered by a rebuild run again on what is there.

The node keeps nothing of a run that has ended: once the server holds its outcome, the node removes the VM, its disk and its log files (after the shipper has read them). The workload's record stays on the server with its status and exit code, and its logs stay in the organization's log store under its id, until the record is deleted (see Workload retention). A microservice restarted from the console gets a fresh VM from the project's current deployment.

Node health

Some of what a node promises can stop being true while it runs, and each failure is silent: a data root that fell back to a filesystem without project quotas keeps running workloads and stops capping their disks, and a firewall chain that was flushed keeps running workloads and stops hiding the cloud metadata endpoint from them.

Each node re-checks these and sends what it can no longer guarantee with its heartbeat, so a node's liveness and its fitness arrive in one call. A node reporting problems reports itself DRAINING, so the orchestrator places no further workloads on it, and ONLINE once it reports none. The workloads already there keep running — a node that stopped enforcing a limit is unfit to take on more, not required to drop what it has.

VmNode.state carries the contract: state.desired is { phase: ONLINE } from the moment the node registers, state.observed is what the node last reported, ONLINE or DRAINING, and state.reconciled is whether the two agree with nothing standing beside the node's word. healthMessage on the node is why it drains, or null when it is fit. Placement selects on state.reconciled, so a node takes workloads exactly when it reports itself fit, the orchestrator can reach it, and its removal has not been asked for; the reservation that follows the pick checks the same field in the shard operation that takes the room, so a node that left its desired state between the two grants nothing and placement picks again. The system console shows the reason on the node.

A node that loses its connection to the server — a server restart, a network partition — reconnects on its own and is reachable again with its next heartbeat. The reconcile master calls the node's worker whenever its next heartbeat would be overdue: a node silent past heartbeatTimeoutSeconds is marked with the condition NODE_UNREACHABLE on its state.conditions, whatever it reported last, and so is every workload whose run is still open on it — starting, running, or stopping. Every such mark, and every status a workload's run passes through, is entered in the kinotic_watch_event data stream with what caused it, and VmNodeService.findHistory and WorkloadService.findHistory read a node's or a workload's entries back, newest first; the system console shows them on the node's page and on the workload's History tab, and its Change log page lists every record's entries across the platform, newest first, through WatchEventService.findAll. The workload record keeps the last status the node reported and its room on the node, because a VM on the far side of a partition is still running; the node's next report of the workload is applied whatever it says, clearing the condition, and a report of a state earlier than the record's is otherwise ignored, since a run only moves forward: the write of a run's status declines one earlier than the record's in the same shard operation, so a write made from a read that a later write overtook leaves the run where it is. Only a heartbeat or a registration writes lastSeen; a state or allocation change leaves it as it is, so it never hides a heartbeat from the timeout, and the mark is conditional on the lastSeen the worker read, so a heartbeat landing between that read and the mark leaves the node unmarked. A call to the node's vm-manager that cannot be delivered is an earlier sign: the orchestrator checks the vm-manager's registration at once and, finding none, marks the node NODE_UNREACHABLE the same way, so nothing is placed on it while the heartbeat timeout runs; its next heartbeat clears the mark. A start or stop the vm-manager took and never answered leaves its workload marked NODE_UNREACHABLE too, since the node may have carried it out, and a microservice whose VM is in that state keeps DEPLOYED in the console with the reason beside it.

Deregistering a node is intent: deregisterNode refuses a reachable node that still runs workloads, and otherwise writes state.deletionRequested, so nothing more is placed on the node, and the node's worker finalizes it — every run still open on an unreachable node is recorded FAILED, which is the operator's word that the node is not coming back, and the record goes with its capacity ledger. Every workload names the deployment it belongs to in state.parent, so a run that ends, on any node, reaches the worker of that deployment.

A workload's diskSizeMb caps the guest rootfs, which is sparse and grows up to that cap. Host directories a workload mounts through volumeMounts are not part of it — their contents are bound from the host filesystem and are not counted against the node's disk.

On BOXLITE nodes a workload may declare at most one entry in volumeMounts: the node mounts a log directory into every workload's VM, and boxlite supports two mounts per VM in total, so a second declared mount is rejected when the workload is started. CLOUD_HYPERVISOR nodes impose no such limit and capture logs without a mount.

Project deployment

The system server deploys a project on every push to its repository's default branch (see Push to Deploy). The pipeline runs a short-lived sync VM per deployment and a short-lived SBOM VM when the project's dependencies changed, which connect to the management server, and one long-lived runtime VM per microservice, which connects to the app server, configured under kinotic.systemApi.deployment.*:

The VMs are the same size in every environment; what differs between nodes is how many fit. The sync VM gets half a core, 2 GB of memory and a 2 GB root filesystem, where the package manager keeps its cache, with the project checkout mount (clone plus installs) capped at 4 GB; each runtime VM, and each site publish or removal VM and SBOM VM, gets half a core, 1 GB of memory and a 512 MB root filesystem — the checkout is a read-only mount, so the root holds only what the microservice writes. The disk caps are enforced where the node's filesystem carries project quotas.

The pipeline also requires the GitHub module (it mints the short-lived fetch token from the org's App installation) and at least one online node advertising a workload data directory — the project's checkout lives at <workloadDataDir>/projects/<projectId> on the node that first deployed it, mounted read-write into the sync VM and read-only into the runtime VMs.

Application API hosts

Each application has an API host of its own, its <organizationId>--<applicationId> label under a platform domain, and every UI a deployment builds is handed its application's host to connect to (see the UI build contract). The app server serves every application at its host: its login and OAuth routes take the application from the host a request is addressed to and answer 404 on any other host, and it signs an application's users in only from pages that are the application's own UIs. Each URL a browser reaches is one setting, in the module every server that reads it loads. The application hosts are kinotic.domain.appApiBaseUrl: the app server serves them, the system server hands each UI build its application's host to connect to, and the management server builds the callback URLs an organization registers with its own identity providers. The portal is kinotic.domain.portalBaseUrl: where the management server's browser flows return and where the platform's emailed links open. The management server's API is kinotic.managementApi.apiBaseUrl, read only by the servers that load management-api. The Helm chart sets all three from the same paths in its values:

UI sites

A deployment of a commit that contains a UI publishes it into the platform's sites storage account, one per environment, and it is served at its own hostname under the organization's subdomain of the sites domain, kinotic.domain.publishedSites.domain below, by the Cloudflare sites Worker. The account is configured under kinotic.systemApi.uiDeployment.*:

The endpoint is validated at boot, like the GitHub App settings, so an environment without Azure storage still sets it, to a placeholder. In the Azure deployment terraform creates the account and the Cloudflare side and passes the account and domain to the server (see the deployment guide); the local profile of the contributing guide names a developer's own. The server never reads or writes the account itself: it holds Storage Blob Data Contributor on it only to sign the URLs, scoped to one site's directory, that the publish and removal workloads act through. The sites Worker reads the account over its blob endpoint with a read-only SAS on the container, which terraform mints and binds to the Worker as a secret; the container is private, so a site's files are reachable only through the Worker.

The sites domain, and the edge each organization's subdomain of it needs a DNS record at, are configured under kinotic.domain.publishedSites.*. The platform creates the record when the organization is created and again before its first site is published:

Find-my-organizations challenge

The landing page's "Email me my organizations" form posts to a route open to anyone that sends an email on every call, so the management server requires a Cloudflare Turnstile token with each request and verifies it with the site's secret, under kinotic.managementServer.turnstile.*:

The development and dev-server profiles use Cloudflare's always-passing test site, so the form works with no site of one's own.

LLM configurations

The large language models the platform's services call are configured at runtime rather than in a property file: a platform operator adds them on the system console's Platform → LLM configurations page, and a service selects one by its name through LlmConfigurationService.findByName, published in the system-api zone.

The API key is given on save, LlmConfigurationService.save(configuration, apiKey), and held through SecretStorageService in the platform's scope, keyed by the configuration's id; a save without one keeps the key held, and deleting the configuration deletes its key. findApiKey(id) reads it back for a service that calls the model for an organization's user, acting as that user's OrganizationParticipant; SYSTEM participants, the system console's operators among them, cannot read it.

Without Azure storage

An environment without the sites storage account, such as development and the tests, turns off everything that uses it with one property:

The development profile and the tests set it; the local profile of the contributing guide turns it off against a developer's own subscription.

Workload storage and log limits

A workload writes to two places the node must bound: the directories it mounts, and its own log output.

sizeLimitMb applies to writable mounts; a mount declared readOnly is already unwritable. It is enforced by an XFS project quota, so the host filesystem refuses the write once the cap is reached whatever the workload does — which requires the mount to live on an XFS filesystem mounted with prjquota, on a node running either provider. The two providers differ only in what a node that cannot enforce a cap does with it. A CLOUD_HYPERVISOR node is provisioned for quotas, so it refuses the workload and reports the condition on its heartbeat until it is fixed. A BOXLITE node may have no project quotas at all — on macOS it cannot have them — so it enforces the cap wherever the filesystem holding the mount carries them, exactly as a CLOUD_HYPERVISOR node does, and only runs a mount uncapped when that filesystem has none; it warns at startup when its workload data directory is on such a filesystem. A node that had project quotas and lost them reports it on its heartbeat and stops taking workloads, whichever provider it runs. A workload's logs occupy at most maxSizeMb * (maxFiles + 1), and the limits are applied by the VM provider rather than by the workload, so a workload cannot raise its own ceiling.

Workload retention

A workload's record is the only link to its logs, so nothing deletes one as a side effect: destroying a workload removes its VM and its disk from the node and records the run STOPPED, and the record and its logs stay. Deleting a workload, from the console or through WorkloadOrchestrationService.deleteWorkload, is the one way either goes: it deletes every log line the workload wrote from the organization's Loki tenant (the platform's own, kinotic-system, for a platform workload), then the record, and refuses a run still open on its node — stop or destroy it first. The retention sweep, one cluster singleton beside the reconcile master, runs hourly and deletes the records of runs that ended longer ago than the retention period the same way, a page of a hundred at a time until none is left.

Configured under kinotic.systemApi.workload.*:

The Loki deletion API needs the compactor's retention and a delete request store, which values-loki.yaml and values-loki-azure.yaml (the object store on Azure) configure, as does the compose file; without them a delete fails and the record stays for the next sweep. Loki's queriers filter a deleted range out of query results once they load the request, at their first query for the tenant or their five-minute reload, at a cost on every query in the tenant until its compactor applies the request, which the configured cancel period of one hour bounds; the sweep asks once per organization for every workload in a page, so a busy platform adds one pending request per organization per hour.

Trace logging

Trace logging prints every STOMP frame the gateway reads and writes, every service invocation the invoker dispatches, and the arguments each invoked method received. That is what makes it useful during development and what makes a high-frequency service — a heartbeat, a poll — drown out everything else in the log.

kinotic.traceLog names the CRIs that stay out of it, and the ones that stay in:

kinotic:
  traceLog:
    excludes:
      - srv://system-api~org.kinotic.system.api.services.workload.VmNodeOrchestrationService/*
      - srv://system-api~org.kinotic.system.api.services.KinoticClusterInfoService/findClusterInfo

Each entry is matched against the fully qualified CRI with Ant wildcards — * matches within one segment, ** across segments — so a trailing /* covers every method of a service and a raw CRI covers just that one method. A scoped invocation carries its scope in the CRI, so a pattern that must cover one needs the scope wildcard as well: srv://*@system-api~com.acme.NodeService/*.

An include wins over an exclude that also matches, which turns the filter around: exclude everything and name what is worth watching.

kinotic:
  traceLog:
    excludes:
      - '**'
    includes:
      - srv://app.acme-org.orders-app~com.acme.OrderService/**

Patterns are matched against every destination but one. A reply comes back on a destination belonging to the client connection rather than naming what was invoked, so it is never matched — an excluded request is marked as it enters the gateway and its reply is dropped from the log along with it. An excluded CRI therefore drops the whole exchange: the request frame, the invocation and its arguments, the values a streaming result emits, and the reply frame the client receives. The patterns are consulted only while trace logging is enabled, and cost nothing at any other level.

kinotic.traceLog is what a node starts with. The system console's Logging dialog, on each node in the Cluster page's server node table, whichever of the three servers it runs, edits the patterns on a running node the same way it sets log levels — see Observability. Those edits last until the node restarts, which returns it to its configured kinotic.traceLog.