Configuration

Overview

Kinotic OS is configured through a combination of application properties, environment variables, and Helm chart values. Key configuration areas include:

• Server configuration — Port bindings, cluster discovery, and runtime settings
• Elasticsearch settings — Connection URLs, index defaults, and replica configuration
• Authentication — OIDC provider configuration, session management, and security policies
• Environment variables — Runtime overrides for containerized deployments
• Helm values — Kubernetes-specific configuration for resource limits, replicas, ingress, and TLS

Loki, Tempo, and Mimir

The server reads workload logs from Grafana Loki, and workload traces and metrics from Grafana Tempo and Mimir (see Observability). Each runs multi-tenant, and the server queries the tenant the caller may see.

In docker-compose the server points at http://loki:3100, http://tempo:3200, and http://mimir:9009; the Helm chart defaults Loki to http://loki.observability.svc:3100 and deploys no Tempo or Mimir yet.

VM provider

Each vm-manager node runs every workload on one VM provider, chosen by the node rather than by the workloads it is given. The node reports which one it runs when it registers, and the orchestrator surfaces it on the node in the system UI.

CLOUD_HYPERVISOR needs the node's Docker daemon to register a kata-clh runtime and to keep its data root on an XFS filesystem mounted with prjquota. Without the quota support the daemon refuses any workload declaring a diskSizeMb, since that is what caps a container's rootfs.

BOXLITE keeps box records and every guest's rootfs disk under one directory. That directory is the filesystem workload disks grow into and the one the node reports its totalDiskMb from, so a node taking workloads of any size wants it on a data volume rather than the root filesystem under the service account's home.

Workload volume mounts

On every node, whichever provider it runs, a workload volume mount must live under the node's workload data directory — mounts are bound with the vm-manager's authority, so this boundary is what keeps a workload spec from mounting an arbitrary host directory. The node creates missing directories for writable mounts, refuses a read-only mount of a directory that does not exist, and reports the directory to the server at registration so deployment flows compose host paths under it. On a CLOUD_HYPERVISOR node the directory should sit on an XFS filesystem with prjquota, since that is what enforces a writable mount's sizeLimitMb; a node whose directory cannot enforce it reports the problem and stops taking workloads.

The default sits under the home directory of the account the vm-manager runs as, so a node creates workload directories with the authority it already has. A node taking real deployments sets it to a path on a data volume — on a CLOUD_HYPERVISOR node, the XFS filesystem whose project quotas enforce sizeLimitMb, which is not where the default lands.

Workload images

A workload's image is pulled before every start when its reference floats — no tag, or latest — so a node runs what the registry holds now rather than the copy it pulled first. A reference pinning a tag other than latest or a digest names one immutable image, and is pulled only when the node does not have it yet. Both providers apply the rule, which is the one Kubernetes uses as its default imagePullPolicy. A BOXLITE node resolves a floating tag to its current digest through the registry itself, anonymously, so its images must be public; a CLOUD_HYPERVISOR node has the container runtime pull the tag.

Workload egress

A CLOUD_HYPERVISOR node denies workload egress by default and permits each workload only what its policy allows. The node's firewall carries the denial; the vm-manager writes one exception per allowed destination when a workload starts, and removes them when it stops.

Every destination comes from the workload's own network.allowedHosts, including the api-gateway — the server sets it there, because only the server knows where the gateway is. The node adds the destinations the workload cannot know: the resolver it was given, and its own OTLP endpoint for a workload that elects telemetry (see Workload Traces and Metrics).

network.allowedHosts entries are IPv4 addresses or CIDRs, not hostnames. Egress rules match addresses, and a hostname resolved once when the workload starts goes stale as soon as the target moves — so a name is rejected rather than resolved. A destination in the same virtual network is best expressed as the subnet's CIDR, which is fixed when the subnet is created.

An empty list means the same on a BOXLITE node. Only the accepted form of an entry differs between providers — addresses and CIDRs on CLOUD_HYPERVISOR, hostnames on BOXLITE.

mode: DISABLED leaves no interface to publish a port on or to reach the node's OTLP endpoint through, so a node refuses a workload declaring it together with portMappings or telemetry — on either provider — rather than starting one whose published ports or telemetry cannot work.

A node that does not deny egress by default cannot honour a declared allowlist, so a workload declaring one is refused there rather than started with access it was supposed to be denied.

A node denies every workload the cloud metadata endpoint (169.254.169.254) and the Azure WireServer's control ports (168.63.129.16), which is what keeps a guest from reading the host's own identity. A workload may still be granted one, by naming the address exactly:

An entry that names a protected address is placed above the node's own rules so it overrides them; every other entry is placed below, so a range containing one grants everything in it except that address. Only a system participant can reach the service that carries a workload's policy, so naming an address outright is a decision the platform made — a range that merely contains one is not, and does not become one by accident.

A provider needs its host prepared for it, so the setting follows the node's provisioning: a developer machine keeps the default, and a node built for Kata is configured for CLOUD_HYPERVISOR. An unrecognised value stops the vm-manager at startup rather than falling back, so a node never registers running something other than what it was configured for.

Node capacity

Each vm-manager node reports its capacity when it registers with the orchestrator. A workload is placed on the first online node whose unallocated capacity covers the workload's vcpus, memoryMb, and diskSizeMb.

Node health

Some of what a node promises can stop being true while it runs, and each failure is silent: a data root that fell back to a filesystem without project quotas keeps running workloads and stops capping their disks, and a firewall chain that was flushed keeps running workloads and stops hiding the cloud metadata endpoint from them.

Each node re-checks these and sends what it can no longer guarantee with its heartbeat, so a node's liveness and its fitness arrive in one call. A node reporting problems moves to DRAINING, so the orchestrator places no further workloads on it, and back to ONLINE once it reports none. The workloads already there keep running — a node that stopped enforcing a limit is unfit to take on more, not required to drop what it has.

VmNode.status carries both parts: status.type is ONLINE, DRAINING, or OFFLINE, and status.healthMessage is why, or null when the node is fit. The system console shows the reason on the node.

A node that loses its connection to the server — a server restart, a network partition — reconnects on its own and is ONLINE again with its next heartbeat. The orchestrator marks it OFFLINE after heartbeatTimeoutSeconds without one.

A workload's diskSizeMb caps the guest rootfs, which is sparse and grows up to that cap. Host directories a workload mounts through volumeMounts are not part of it — their contents are bound from the host filesystem and are not counted against the node's disk.

On BOXLITE nodes a workload may declare at most one entry in volumeMounts: the node mounts a log directory into every workload's VM, and boxlite supports two mounts per VM in total, so a second declared mount is rejected when the workload is started. CLOUD_HYPERVISOR nodes impose no such limit and capture logs without a mount.

Project deployment

The system server deploys a project on every push to its repository's default branch (see Push to Deploy). The pipeline runs a short-lived sync VM per deployment and one long-lived runtime VM per microservice, configured under kinotic.systemApi.deployment.*:

The pipeline also requires the GitHub module (it mints the short-lived fetch token from the org's App installation) and at least one online node advertising a workload data directory — the project's checkout lives at <workloadDataDir>/projects/<projectId> on the node that first deployed it, mounted read-write into the sync VM and read-only into the runtime VMs.

Organization storage

A deployment of a commit that contains a UI needs storage to publish it to. The platform keeps one storage account per organization, provisioned when the organization is created and configured under kinotic.systemApi.organizationStorage.*:

The required settings are validated at boot, like the GitHub App settings, so an environment that disables the provisioner still sets them, to placeholders; nothing reads them while it is disabled. In the Azure deployment, terraform creates the resource group, subnet and private DNS zone and passes their ids to the server (see the deployment guide). The development profile disables the provisioner, points at a local Azurite and disables private endpoints; a developer who wants the real path adds the local profile with their own subscription, as the contributing guide describes.

UI sites

Each published UI is served from its own site, a hostname under a platform domain fronted by Azure Front Door, configured under kinotic.systemApi.uiDeployment.*:

Like organization storage, the required settings are validated at boot, and an environment that disables the provisioner sets them to placeholders. In the Azure deployment terraform creates the profile and endpoint and passes them to the server; the development profile disables the provisioner, and the local profile of the contributing guide enables it against a developer's own profile. Front Door reads an organization's storage as the profile's managed identity, which terraform grants Storage Blob Data Reader on the group every organization's account is created in; the account's public network is open to it, with anonymous access off.

Workload storage and log limits

A workload writes to two places the node must bound: the directories it mounts, and its own log output.

sizeLimitMb applies to writable mounts; a mount declared readOnly is already unwritable. It is enforced by an XFS project quota, so the host filesystem refuses the write once the cap is reached whatever the workload does — which requires the mount to live on an XFS filesystem mounted with prjquota, on a node running either provider. The two providers differ only in what a node that cannot enforce a cap does with it. A CLOUD_HYPERVISOR node is provisioned for quotas, so it refuses the workload and reports the condition on its heartbeat until it is fixed. A BOXLITE node may have no project quotas at all — on macOS it cannot have them — so it enforces the cap wherever the filesystem holding the mount carries them, exactly as a CLOUD_HYPERVISOR node does, and only runs a mount uncapped when that filesystem has none; it warns at startup when its workload data directory is on such a filesystem. A node that had project quotas and lost them reports it on its heartbeat and stops taking workloads, whichever provider it runs. A workload's logs occupy at most maxSizeMb * (maxFiles + 1), and the limits are applied by the VM provider rather than by the workload, so a workload cannot raise its own ceiling.

Trace logging

Trace logging prints every STOMP frame the gateway reads and writes, every service invocation the invoker dispatches, and the arguments each invoked method received. That is what makes it useful during development and what makes a high-frequency service — a heartbeat, a poll — drown out everything else in the log.

kinotic.traceLog names the CRIs that stay out of it, and the ones that stay in:

kinotic:
  traceLog:
    excludes:
      - srv://system-api~org.kinotic.system.api.services.VmNodeOrchestrationService/*
      - srv://system-api~org.kinotic.system.api.services.KinoticClusterInfoService/findClusterInfo

Each entry is matched against the fully qualified CRI with Ant wildcards — * matches within one segment, ** across segments — so a trailing /* covers every method of a service and a raw CRI covers just that one method. A scoped invocation carries its scope in the CRI, so a pattern that must cover one needs the scope wildcard as well: srv://*@system-api~com.acme.NodeService/*.

An include wins over an exclude that also matches, which turns the filter around: exclude everything and name what is worth watching.

kinotic:
  traceLog:
    excludes:
      - '**'
    includes:
      - srv://app.acme-org.orders-app~com.acme.OrderService/**

Patterns are matched against every destination but one. A reply comes back on a destination belonging to the client connection rather than naming what was invoked, so it is never matched — an excluded request is marked as it enters the gateway and its reply is dropped from the log along with it. An excluded CRI therefore drops the whole exchange: the request frame, the invocation and its arguments, the values a streaming result emits, and the reply frame the client receives. The patterns are consulted only while trace logging is enabled, and cost nothing at any other level.

kinotic.traceLog is what a node starts with. The system console's Logging dialog, on each node in the Cluster page's server node table, edits the patterns on a running node the same way it sets log levels — see Observability. Those edits last until the node restarts, which returns it to its configured kinotic.traceLog.