Kasm Workspaces observability

Kasm Workspaces ships a built-in OpenTelemetry (OTel) stack that provides centralized, structured telemetry across all platform services. Administrators can monitor session activity, diagnose failures, and correlate events across logs, traces, and metrics without deploying any external agents.

What ships in the platform

Every Kasm Workspaces installation includes three built-in observability components.

• An OTel Collector (kasm_otel_collector) that receives telemetry from all services on the node and routes it to storage.
• A ClickHouse database (kasm_clickhouse) that stores logs, traces, and metrics with configurable per-signal retention periods.
• OTel SDK instrumentation in every Kasm role service (API server, manager, agent, proxy services, and RDP gateway services).

In a single-server (all) installation, ClickHouse is bundled on the node. In a multi-server deployment, ClickHouse runs on its own dedicated clickhouse role node — a required role, the same as the Database Server — which stores telemetry over TLS-only connections and forwards no telemetry of its own. See Deploy and manage observability.

In a distributed deployment, each remote node runs its own collector. That collector forwards signals to the central collector on the app node.

The three telemetry signals

Kasm Workspaces emits three OpenTelemetry signal types: logs, traces, and metrics.

Logs

Log records capture structured events from each service. Every record carries the service name, log level, timestamp, and a set of Kasm-specific attributes. For events tied to an active workspace session, the kasm.kasm.id attribute identifies the session. Logs reach three destinations simultaneously: the service console, a JSON file on disk, and ClickHouse via OTLP.

Traces

Distributed traces record the execution path of a request across one or more services. Each trace uses the W3C traceparent header for span propagation, so child spans from downstream services link automatically to the originating request. The browser is the root of the trace for user-initiated requests, and server spans appear as children. For details, see Browser as trace root.

Metrics

Metrics capture counters, gauges, and histograms from each service. Counters and gauges use a 15-second DELTA export interval. Resource gauges on agent-role services use a 30-second interval. For a complete catalog of metric names and their attributes, see Observability metrics reference.

Log channels

Each Kasm role service emits log records to three channels simultaneously.

Channel	Format	Destination	Rotation policy
Console	Colored text	Standard output of the service container	Not applicable
File	JSON	/opt/kasm/current/log/api.log and agent.log	Daily or 10 MB, 20 files kept
OTLP	Protobuf	Collector on port 4318 (HTTP) then ClickHouse kasm_otel.otel_logs table	ClickHouse TTL governs retention

The file channel is enabled by default. Set KASM_LOG_FILE_ENABLED=false to disable it in environments such as Kubernetes where a container log collector manages persistence.

Log file rotation

The API server writes api.log and the agent writes agent.log. The manager runs inside the API server process, so its records appear in api.log. Each file rotates on a time trigger and a size trigger, and whichever fires first wins. For the s, m, h, and d units, every rotation, including a size rotation, restarts the time window from that moment, so a busy service can rotate on size before the time trigger is ever reached. midnight and w0 to w6 keep a fixed schedule. The two triggers work as follows:

• Time. By default the file rotates at midnight UTC, because the service containers do not set a time zone. The rotated file takes a date suffix, for example api.log.2026-10-08. The schedule can be changed to any number of seconds, minutes, hours, or days, or to a weekly rotation on a fixed weekday. Finer units produce a finer timestamp suffix, such as api.log.2026-10-08_14 for hourly rotation.
• Size. When the active file reaches 10 MB by default, it rotates immediately. A size rotation uses the same timestamped name as the current time window. If that backup already exists, the existing backup is renamed with a counter (api.log.2026-10-08.1, then .2, and so on) before the new backup takes the plain name. A burst of traffic therefore produces several files for one window and never overwrites earlier records. The plain timestamped file is always the newest backup of its window.

Retention is one pool. Twenty rotated files are kept per service by default, counted across both triggers and including counter-suffixed files. The oldest files are deleted at each rotation. With the defaults, a service holds at most about 200 MB of rotated logs on disk. A lower size threshold or backup count reduces that ceiling.

A size-only policy (time trigger set to a long interval) bounds disk use predictably on busy deployments. A time-only policy (size trigger disabled) keeps one file per period, which suits log shippers that expect a fixed naming pattern. The default combines both so that quiet days still produce a daily file and busy days cannot fill the disk.

To change the schedule, size threshold, or retained count, see Configure log file rotation. For the variables and their accepted values, see Observability configuration.

nginx proxy logs

The nginx proxy service is instrumented differently from role services. nginx writes access logs in JSON format directly to /opt/kasm/current/log/nginx/access_json.log. The OTel Collector reads this file using a filelog receiver and forwards each record to ClickHouse with service.name=kasm-proxy.

Severity is derived from the HTTP status code of each request.

HTTP status range	OTel severity
2xx	INFO
3xx	DEBUG
4xx	WARN
5xx	ERROR

nginx also exposes a stub_status endpoint that the collector scrapes every 30 seconds to produce connection and request count metrics. For the full list of nginx metrics, see nginx proxy metrics.

PostgreSQL database logs

The Database Server is instrumented the same way as the nginx proxy: PostgreSQL has no OTLP exporter, so the OTel Collector reads its logs from disk instead of receiving them over OTLP. postgresql.conf writes structured jsonlog output alongside the existing stderr text log, and the collector tails that JSON file and forwards each record to ClickHouse with service.name=kasm-db.

Severity is derived from PostgreSQL's error_severity field.

PostgreSQL severity	OTel severity
PANIC, FATAL	FATAL
ERROR	ERROR
WARNING	WARN
LOG, NOTICE, INFO	INFO
DEBUG1–DEBUG5	DEBUG

The collector also renames select fields to the kasm.db.* attribute namespace, including the database name, connecting user, application name, and SQL state code.

The collector additionally scrapes PostgreSQL directly every 30 seconds using a dedicated, read-only kasm_otel_monitor role to produce database metrics. For the full list of PostgreSQL metrics and the role's grants, see PostgreSQL database metrics and PostgreSQL monitoring role.

Telemetry topology

Single-node deployment

On a single-node install, all services and the collector run on the same host. Services emit OTLP/HTTP to the collector on the Docker-internal port 4318. The collector writes all signals to the local ClickHouse instance. No telemetry traffic crosses a network boundary.

The following diagram shows the signal flow on a single node.

Description: Application services send telemetry over OTLP/HTTP to the local collector on port 4318. The nginx proxy and the PostgreSQL database each contribute logs and metrics via a file-based or scrape-based path read by the collector, rather than emitting OTLP themselves. The collector batches all signals and writes them to ClickHouse.

Key takeaways:

• All traffic stays within the Docker internal network.
• The collector is the single ingress point for ClickHouse writes.
• nginx and PostgreSQL both use collector-side receivers (filelog and scrape) rather than the OTLP SDK.

Distributed deployment

In a distributed deployment, every remote node (additional agent nodes, RDP gateway nodes, and the db role node) runs a lightweight collector. Remote collectors forward all signals over OTLP gRPC to the central collector on the app node. Port 4317 (gRPC) is the only port that must be open between application nodes. The app role never bundles ClickHouse — it connects over the network to a dedicated clickhouse role node (or an external ClickHouse instance), the same way it connects to the Database Server.

This inter-collector hop is bearer-token authenticated and TLS-encrypted by default. Customers who install the app/all node with their own CA-signed certificate can additionally opt in to certificate verification on this hop — see Configure custom TLS certificates and 4317 inter-collector TLS verification.

The following diagram shows the cross-node signal flow.

Description: Each remote node, including the db role node, aggregates its own telemetry in a local collector before forwarding it to the central collector over gRPC (bold arrows). The central collector on the app node is the only component that writes to ClickHouse, which runs on its own dedicated clickhouse role node (or an external instance) — the app node has no bundled ClickHouse of its own.

Key takeaways:

• Port 4317 (gRPC) must be open between application nodes, including the db role node.
• Remote collectors provide local buffering, which reduces the impact of transient connectivity issues between nodes.
• All ClickHouse writes originate from the central collector on the app node.
• The app node needs outbound access to the ClickHouse node on TCP 8443 (HTTPS reads for UI reports) and 9440 (secure native writes by the collector), the same as it needs access to the Database Server.
• The db role node's collector reads PostgreSQL's log file and scrapes PostgreSQL metrics locally, then forwards both upstream over the same gRPC hop as any other remote node.

Browser as trace root

When a user opens a Kasm Workspaces session in a browser, the browser OTel SDK instruments outgoing XHR and fetch calls automatically. The resulting browser span becomes the root of the distributed trace, and server-side spans from Kasm application services appear as children in the same trace. Automatic clock correction handles browser clock drift, keeping trace timelines coherent even when the browser clock differs from the server clock.

The nginx proxy also emits trace spans for every proxied request, so nginx spans appear as parents of the application service spans they route to.

Session correlation

Every telemetry signal that relates to an active workspace session carries the attribute kasm.kasm.id, which contains the unique session identifier. This attribute is present on log records, trace spans, and metrics emitted during the session lifetime. Filtering by kasm.kasm.id in ClickHouse returns all signals across all services for a single session, enabling full cross-service correlation.

Degradation and no-database mode

If ClickHouse becomes unavailable, the platform logs a warning and skips telemetry writes. All business functionality (session provisioning, user authentication, workspace delivery) continues without interruption. The collector's sending queue provides approximately five minutes of buffering against transient ClickHouse unavailability. Records emitted after the queue is full are dropped.

When Kasm Workspaces is installed with the --no-clickhouse flag and no dedicated or external ClickHouse replacement is configured, the collector starts in debug exporter mode. In this mode, the collector writes all received signals to its own standard output and persists no data. This mode is intended for environments where telemetry persistence is managed externally or is not required.

Integrating with external tools

ClickHouse is the built-in storage and query backend. Administrators can forward all signals in parallel to any OTLP-compatible backend using the --otel-forward-endpoint flag at install or upgrade time. Supported targets include Grafana Cloud, Splunk, and any receiver that accepts OTLP over HTTP or gRPC.

For all configuration options, see Observability configuration reference.