Version: Developer

Deploy and manage observability

Overview

Kasm Workspaces 1.20.0 ships with a built-in observability stack that collects logs, traces, and metrics from every service. The stack consists of an OpenTelemetry (OTel) Collector and a ClickHouse database, both deployed automatically on standard installations. This guide walks administrators through deployment options, distributed configuration, and operational controls for the observability stack.

Prerequisites

Root access to the Kasm Workspaces host or hosts.
Kasm Workspaces 1.20.0 or later.
For distributed deployments: network connectivity between nodes and the ability to configure firewall rules.

Solution approach

This guide progresses through the following phases:

Understand the default deployment and opt-out behavior.
Deploy the observability stack for single-node and distributed environments.
Customize the configuration for external components, TLS, and retention.
Run ClickHouse on its own dedicated node, required for multi-server deployments.
Control individual signals and horizontal-scaling instance identity.

Detailed steps

Understand the default deployment and opt-out behavior

Observability is enabled by default. On a single-server all installation, the installer deploys kasm_otel_collector and kasm_clickhouse automatically, and no additional flags are required. On an app node, the installer deploys only kasm_otel_collector; ClickHouse is never bundled with the app role. The app role must connect to a dedicated (--role clickhouse) or external ClickHouse instance, exactly like the Database Server — see Deploy in a distributed environment below.

To opt out of the observability stack at install time, pass --skip-otel:

sudo bash install.sh --role all --skip-otel

This choice is persistent. Subsequent upgrades keep the observability stack uninstalled unless --enable-otel is explicitly passed to the upgrade script:

sudo bash upgrade.sh --enable-otel

Deploy on a single node

For a single-node deployment using the all role, run the installer without any observability flags:

sudo bash install.sh --role all --cfg /path/to/config.yaml

The installer automatically performs the following steps:

Generates an OTel bearer token (32 characters) and a ClickHouse password (20 characters).
Deploys kasm_otel_collector (local variant) and kasm_clickhouse.
Configures all Kasm services with the collector endpoint.
Creates the kasm_clickhouse Docker volume.

Deploy in a distributed environment

A distributed deployment uses one db node, one app node, and one or more agent nodes. The central OTel Collector runs on the app node. Remote collectors on the db and agent nodes forward telemetry to it over TCP 4317.

Step 1. Install the database node:

sudo bash install.sh --role db \
  --otel-token "YOUR_SHARED_TOKEN" \
  --otel-collector-hostname "app-node-hostname"

The db role also generates the kasm_otel_monitor PostgreSQL role and password used by the postgresql metrics receiver, and prints the password at the end of installation. Save it — the app node needs it in the next step.

Step 2. Install the app node:

The app role never bundles ClickHouse. It must connect to a dedicated (--role clickhouse) or external ClickHouse instance, the same as it must connect to a database — pass the ClickHouse connection flags along with the database flags, or pass --no-clickhouse to run without ClickHouse (see Enable no-database mode):

sudo bash install.sh --role app \
  --otel-token "YOUR_SHARED_TOKEN" \
  --clickhouse-endpoint "tcp://ch-node-hostname:9440?secure=true&skip_verify=true" \
  --clickhouse-http-endpoint "https://ch-node-hostname:8443" \
  --clickhouse-username "kasm_otel" \
  --clickhouse-password "YOUR_CH_PASSWORD" \
  --clickhouse-reader-username "kasm_otel_ro" \
  --clickhouse-reader-password "YOUR_CH_READER_PASSWORD" \
  --db-host "db-node-hostname" --db-password "DB_PASSWORD" \
  --otel-db-monitor-password "OTEL_DB_MONITOR_PASSWORD_FROM_STEP_1"

note

If --otel-db-monitor-password is omitted, install.sh prompts interactively for the kasm_otel_monitor password generated on the db node, the same way it prompts for a missing ClickHouse password.

note

ClickHouse uses two separate credential pairs. The --clickhouse-username/--clickhouse-password pair (default kasm_otel) is the writer the OTel Collector uses to insert telemetry. The --clickhouse-reader-username/--clickhouse-reader-password pair (default kasm_otel_ro) is the reader kasm_api and kasm_manager use for UI reports and the log viewer. Pass both pairs to any app/all node that connects to a dedicated or external ClickHouse instance — omitting the reader credentials breaks UI reports and the log viewer. See Network prerequisite: TCP 8443 and 9440 for the grants each user needs.

If the ClickHouse flags are omitted and --no-clickhouse is not passed, install.sh interactively prompts for the ClickHouse Server hostname and password, exactly like it prompts for the database hostname.

Step 3. Install each agent node:

sudo bash install.sh --role agent \
  --otel-token "YOUR_SHARED_TOKEN" \
  --otel-collector-hostname "app-node-hostname" \
  --agent-registration-hostname "app-node-hostname" \
  --agent-token "MANAGER_TOKEN"

The --otel-token value must be identical across all nodes. The --otel-collector-hostname on remote roles must point to the app or all node.

Database Server role specifics

On db and all nodes, the OTel Collector also collects PostgreSQL logs and metrics directly, the same pattern the nginx proxy uses on the app and all roles.

Volume mount — the installer mounts /opt/kasm/current/log/postgres into the collector container at /var/log/postgres/external (read-only) and tails PostgreSQL's jsonlog output.
postgresql receiver — the collector scrapes PostgreSQL directly every 30 seconds using the dedicated kasm_otel_monitor role, which is granted the built-in pg_monitor role and has no access to application data.

Both are automatic on db/all — the monitor role's password is generated during install and recovered automatically on upgrade, the same as the ClickHouse password. On the app role, since the database is external, install.sh prompts for the existing kasm_otel_monitor password, or accepts it directly with --otel-db-monitor-password (see Step 2 above). For details on the role's grants and the log format, see PostgreSQL monitoring role.

Network prerequisite: TCP 4317

Remote collectors forward telemetry to the central collector on the app or all node over OTLP/gRPC port 4317. Before adding any remote nodes, allow inbound TCP 4317 on the app or all node from the remote nodes' IP addresses or subnet.

Cloud firewall examples:

AWS: Add an inbound rule to the app or all node's Security Group: Protocol TCP, Port 4317, Source = remote nodes' Security Group or CIDR.
Azure: Add an inbound rule to the app or all node's Network Security Group: Port 4317, Source = remote nodes' IP range.
GCP: Add a firewall rule: Allow TCP 4317, target = app or all node, source = remote nodes' tag or CIDR.

caution

Do not open port 4317 to the internet. The endpoint uses bearer token authentication, and its transport is TLS-encrypted by default, but it does not need to be publicly reachable.

Single-node (all) deployments do not require port 4317 open because there are no remote collectors. Docker still publishes the port on the host; block it at the host or cloud firewall on single-node installs if required by your security policy.

Ports 4318 (HTTP), 4316 (nginx proxy), and 8090 (nginx stub_status) are Docker-network only and do not cross node boundaries. No firewall rules are needed for these ports.

Network prerequisite: TCP 8443 and 9440 (dedicated ClickHouse only)

If observability storage is offloaded to a dedicated clickhouse role node (see Deploy ClickHouse on a dedicated node above), the app/all node must reach that node on two additional ports:

TCP 8443 (HTTPS): kasm_api and kasm_manager read telemetry over HTTPS for UI reports and the log viewer.
TCP 9440 (secure native TCP): the OTel Collector on the app/all node writes telemetry over the secure ClickHouse native protocol.

Before adding a dedicated ClickHouse node, allow inbound TCP 8443 and 9440 on the clickhouse node from the app/all node's IP address or subnet.

caution

Do not open ports 8443 or 9440 to the internet. Both interfaces are TLS-only but are not intended for direct client access.

These two ports also correspond to two separate ClickHouse users, following least privilege. The writer (--clickhouse-username, default kasm_otel) uses port 9440 and needs SELECT, INSERT, CREATE DATABASE, CREATE TABLE, and CREATE VIEW grants so the collector can write telemetry. The reader (--clickhouse-reader-username, default kasm_otel_ro) uses port 8443 and needs only SELECT, because kasm_api and kasm_manager only query telemetry for reports and the log viewer. A compromised writer credential can append data but cannot rewrite or delete already-retained records.

These ports do not apply to a single-server all installation, where ClickHouse is bundled and the interfaces remain Docker-internal only. In any multi-server deployment, the app node has no bundled ClickHouse and must reach the ClickHouse node — dedicated or external — on these ports.

Configure custom TLS certificates

The inter-collector gRPC channel (port 4317) uses the same TLS certificates as the nginx proxy, and is bearer-token authenticated and TLS-encrypted by default. Provide custom certificates at install time:

sudo bash install.sh --role all \
  --ssl-public-cert /path/to/cert.pem \
  --ssl-private-key /path/to/key.pem

The installer mounts the certificates into the collector container at /etc/otel/tls.crt and /etc/otel/tls.key. By default this hop is encrypt-only: remote nodes do not verify the central collector's certificate, because Kasm's self-signed default certificate has no Subject Alternative Names (SANs) to check against.

Enable verified TLS on the inter-collector hop

If the app/all node is installed with your own CA-signed certificate instead of Kasm's self-signed default, you can turn on certificate verification for the 4317 hop. This is optional and adds transport verification on top of the bearer-token authentication that is already in place.

Prerequisites:

The app/all node must be installed (or re-installed) with a CA-signed certificate, as shown above.
That certificate's SANs must cover the exact hostname remote nodes use to reach it — the value passed as --otel-collector-hostname on each remote node. For example, if remote nodes use --otel-collector-hostname kasm-app.example.com, the certificate needs kasm-app.example.com as a SAN.

Enable it by passing one of the following flags on each remote node (agent, guac, proxy; no effect on db, which runs no collector container):

Flag	Use case
`--otel-collector-verify-tls`	The `app`/`all` node's certificate is signed by a publicly-trusted CA. Verifies against the collector image's system trust store.
`--otel-collector-ca-cert <path>`	The `app`/`all` node's certificate is signed by a private or corporate CA. Implies `--otel-collector-verify-tls` and additionally trusts the CA certificate at `<path>` — the recommended option in either case.

For example, to install a Connection Proxy node with verified TLS against a corporate CA:

sudo bash install.sh --role guac \
  --api-hostname kasm-app.example.com \
  --public-hostname 192.0.2.20 \
  --registration-token [SERVICE_REGISTRATION_TOKEN] \
  --otel-token [OTEL_TOKEN] \
  --otel-collector-hostname kasm-app.example.com \
  --otel-collector-ca-cert /path/to/corporate-ca.pem

The setting persists across upgrades: upgrade.sh recovers a node's existing verify-tls choice from its current rendered configuration automatically. Pass --otel-collector-verify-tls/--otel-collector-ca-cert explicitly on an upgrade.sh command line to override the recovered value, or --otel-collector-no-verify-tls to durably clear a previously-enabled choice.

Mutual TLS is not available on this hop.

If verification fails after enabling it, see Diagnose TLS handshake failures on the inter-collector hop.

Configure an external collector

Use --otel-collector-endpoint to replace the bundled collector with your own (for example, Grafana Alloy):

sudo bash install.sh --role all \
  --otel-collector-endpoint "http://my-alloy-collector:4318"

On the all role, the bundled ClickHouse remains deployed unless --no-clickhouse is also specified. The app role has no bundled ClickHouse to begin with — see Deploy ClickHouse on a dedicated node below.

Deploy ClickHouse on a dedicated node

Install --role clickhouse to run ClickHouse on its own dedicated host. In a multi-server deployment, the app role has no bundled ClickHouse and must connect to this dedicated instance (or an external one) — this role is required, the same way the Database Server role is. Only a single-server all installation bundles ClickHouse; an all node can optionally be pointed at a dedicated or external ClickHouse instance instead of using the bundled one.

Step 1. Install the dedicated ClickHouse node:

sudo bash install.sh --role clickhouse \
  --clickhouse-password "YOUR_CH_PASSWORD" \
  --clickhouse-reader-password "YOUR_CH_READER_PASSWORD"

This node runs kasm_clickhouse only — no OTel Collector is deployed, and it does not forward telemetry on its own. ClickHouse listens on TLS-only ports published to the network: 8443 (HTTPS) and 9440 (secure native TCP). The installer creates two ClickHouse users — a writer (kasm_otel) and a reader (kasm_otel_ro) — and prints both credential pairs together with the flags to pass to the app node.

Step 2. Point the app node at it:

sudo bash install.sh --role app \
  --clickhouse-endpoint "tcp://ch-host:9440?secure=true&skip_verify=true" \
  --clickhouse-http-endpoint "https://ch-host:8443" \
  --clickhouse-username "kasm_otel" \
  --clickhouse-password "YOUR_CH_PASSWORD" \
  --clickhouse-reader-username "kasm_otel_ro" \
  --clickhouse-reader-password "YOUR_CH_READER_PASSWORD" \
  --db-host "db-node-hostname" --db-password "DB_PASSWORD"

The skip_verify=true parameter is required because no certificate authority is distributed between nodes — the connection is still encrypted, but the app node does not verify the ClickHouse Server's certificate identity. This matches the Database Server's default TLS posture (sslmode=require): encrypted, without certificate verification. If you distribute your own trusted CA to both nodes, replace it with skip_verify=false to require certificate verification.

--clickhouse-password and --clickhouse-reader-password must match the values set during the clickhouse role install. Before continuing, verify the app node can reach the ClickHouse node on TCP 8443 and 9440. See Network prerequisite: TCP 4317 for firewall guidance — the same allow-list pattern applies to 8443 and 9440.

Retention/TTL of telemetry is managed on whichever node hosts ClickHouse — the clickhouse role, or all for single-server deployments. Retention flags passed to the app node have no effect when it points at a dedicated (or external) ClickHouse.

Configure an external ClickHouse instance

Before installing, create two users on the external ClickHouse instance: a writer for collector writes, and a reader for UI reports and the log viewer.

CREATE USER kasm_otel IDENTIFIED BY 'YOUR_CH_PASSWORD';
GRANT SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW ON <database>.* TO kasm_otel;

CREATE USER kasm_otel_ro IDENTIFIED BY 'YOUR_CH_READER_PASSWORD';
GRANT SELECT ON <database>.* TO kasm_otel_ro;

The writer requires the CREATE DATABASE grant because the collector's ClickHouse exporter issues CREATE DATABASE IF NOT EXISTS on startup. The reader only ever runs SELECT queries for reports and the log viewer, so grant it SELECT alone.

Use --clickhouse-endpoint and --clickhouse-http-endpoint together to point to an external ClickHouse instance you manage yourself (outside a Kasm-managed clickhouse role node). Both flags are required; the collector uses the native DSN and the Kasm UI uses the HTTP URL for reports.

sudo bash install.sh --role all \
  --clickhouse-endpoint "tcp://ch-host:9440?secure=true&skip_verify=true" \
  --clickhouse-http-endpoint "https://ch-host:8443" \
  --clickhouse-username "kasm_otel" \
  --clickhouse-password "YOUR_CH_PASSWORD" \
  --clickhouse-reader-username "kasm_otel_ro" \
  --clickhouse-reader-password "YOUR_CH_READER_PASSWORD"

Pass both credential pairs. Omitting the reader flags leaves kasm_api and kasm_manager without a way to authenticate reads, and UI reports and the log viewer fail.

With an external ClickHouse instance, the administrator manages retention TTLs directly. The installer does not alter external databases. By default the connection is encrypted but the server's certificate identity is not verified — the same TLS posture the installer uses for the Database Server (sslmode=require). If your external ClickHouse presents a certificate signed by a CA you distribute to the app/all node yourself, replace skip_verify=true with skip_verify=false to require certificate verification.

Enable no-database mode

When --no-clickhouse is passed without a dedicated or external ClickHouse instance and without a custom collector, the installer replaces the ClickHouse exporters with a debug exporter. The collector starts and logs received signals to stdout but does not persist data. The Kasm platform continues running; telemetry read paths return empty results with a warning. This mode suits environments where storing telemetry data locally is not permitted.

Configure retention

Pass retention flags to the installer or upgrade script. The defaults are 30 days for general logs, 365 days for audit logs, 7 days for traces, and 90 days for metrics. For the full flag list, see Observability configuration.

Audit-class logs — login and logout events, and records with the kasm.retention_class=audit attribute — are automatically retained for the longer audit retention period.

To update retention on an existing installation, re-run the upgrade script with the desired flags. This updates both the ClickHouse TTL and the collector configuration. A failed retention update stops the upgrade with one error line per table and a non-zero exit status. Fix the reported cause, then re-run the upgrade script with the same flags.

Configure sampling

The central collector uses tail-based sampling. The default policy keeps all errors, keeps all slow traces (1000 ms or longer), and retains 10% of remaining traces probabilistically. Adjust the probabilistic rate with --otel-trace-sample-percentage <n>.

The sampling decision window is 30 seconds. Traces longer than 30 seconds — for example, slow VM provisioning operations — may be evaluated on partial span data.

Disable individual signals

Set any of the following environment variables on a service container to suppress specific signals:

OTEL_LOGS_EXPORTER=none     # Disable OTLP log export (file and console logging are unchanged)
OTEL_TRACES_EXPORTER=none   # Disable tracing
OTEL_METRICS_EXPORTER=none  # Disable metrics
OTEL_SDK_DISABLED=true      # Disable all OTLP export

Set KASM_LOG_FILE_ENABLED=false to disable file logging in environments without writable log volumes, such as Kubernetes.

Configure log file rotation

The JSON log files rotate at midnight UTC and whenever a file reaches 10 MB, and 20 rotated files are kept per service. To change the schedule, the size threshold, or the retained count:

Edit /opt/kasm/current/docker/docker-compose.yaml and set the rotation variables in the environment block of the kasm_api service, the kasm_agent service, or both. Set only the variables you want to change. The others keep their defaults. An upgrade writes a new docker-compose.yaml, so set these variables again after each upgrade.
docker-compose.yaml
```
environment:
  KASM_LOG_FILE_WHEN: "h"             # rotate on an hourly schedule
  KASM_LOG_FILE_INTERVAL: "6"         # every 6 hours
  KASM_LOG_FILE_MAX_BYTES: "52428800" # or as soon as the file reaches 50 MB
  KASM_LOG_FILE_BACKUP_COUNT: "48"    # keep the 48 most recent rotated files
```
To rotate on size only, set KASM_LOG_FILE_WHEN to d and KASM_LOG_FILE_INTERVAL to a large value such as 3650. To rotate on time only, set KASM_LOG_FILE_MAX_BYTES to 0.
Restart the services so the new environment takes effect. The variables are read once at process start.
```
sudo /opt/kasm/bin/stop
sudo /opt/kasm/bin/start
```
Confirm the new policy by listing the log directory after the first rotation. Rotated files carry a timestamp suffix in the granularity of the chosen unit, for example api.log.2026-10-08_14 for hourly rotation. A .1, .2, and so on after the timestamp marks an earlier size rotation within the same window.
```
ls -l /opt/kasm/current/log/api.log*
```

The retained count applies across both triggers, so the disk ceiling per service is roughly KASM_LOG_FILE_BACKUP_COUNT multiplied by KASM_LOG_FILE_MAX_BYTES. For the accepted values of each variable, see Observability configuration. For how the two triggers interact, see Log file rotation.

Add attributes to the JSON log files

The JSON log files under /opt/kasm/current/log/ carry six base fields: timestamp, level, logger, module, line, and message. To add OTel log attributes to each line:

Edit /opt/kasm/current/docker/docker-compose.yaml and add KASM_LOG_FILE_EXTRA_FIELDS to the environment block of the kasm_api and kasm_agent services. List the attribute names separated by commas.
docker-compose.yaml
```
environment:
  KASM_LOG_FILE_EXTRA_FIELDS: "kasm.metric_name,kasm.kasm.id,kasm.zone.name"
```
Restart the services so the new environment takes effect.
```
sudo /opt/kasm/bin/stop
sudo /opt/kasm/bin/start
```
Confirm that new lines in /opt/kasm/current/log/api.log carry the fields with underscores, for example kasm_kasm_id.

Names a service never sets are ignored, so one list works for both services. For the attribute names each session event carries, see Session lifecycle events.

Configure horizontal-scaling instance identity

When running multiple replicas, keep OTEL_SERVICE_NAME set to the generic service name. Use OTEL_RESOURCE_ATTRIBUTES to distinguish individual replicas:

OTEL_SERVICE_NAME=kasm-guac
OTEL_RESOURCE_ATTRIBUTES=service.instance.id=guac-2,host.name=node-east-03

This configuration allows aggregating telemetry by service name while still drilling into a specific replica.

For the full flag and environment variable reference, see Observability configuration.

Common troubleshooting steps

No data appearing in ClickHouse. Verify the collector is running with docker ps | grep kasm_otel, then check docker logs kasm_otel_collector for export errors.
Remote node telemetry missing. Confirm that TCP 4317 is open between the remote node and the app or all node, and that the --otel-token value matches on all nodes.
Upgrade leaves OTel uninstalled. If --skip-otel was used at initial install, pass --enable-otel to the upgrade script.

For detailed diagnostic steps, see Troubleshoot observability.

Overview​

Prerequisites​

Solution approach​

Detailed steps​

Understand the default deployment and opt-out behavior​

Deploy on a single node​

Deploy in a distributed environment​

Database Server role specifics​

Network prerequisite: TCP 4317​

Network prerequisite: TCP 8443 and 9440 (dedicated ClickHouse only)​

Configure custom TLS certificates​

Enable verified TLS on the inter-collector hop​

Configure an external collector​

Deploy ClickHouse on a dedicated node​

Configure an external ClickHouse instance​

Enable no-database mode​

Configure retention​

Configure sampling​

Disable individual signals​

Configure log file rotation​

Add attributes to the JSON log files​

Configure horizontal-scaling instance identity​

Common troubleshooting steps​