Kasm Workspaces: Observability migration guide

warning

Breaking changes in 1.20.0 — review all items before starting the upgrade.

1. PostgreSQL logs table removed. The schema migration drops the logs table. Historical log data is not migrated to ClickHouse. The pre-upgrade backup at /opt/kasm/backups/kasm_db_backup.tar contains the final state of the table. Keep this backup if pre-upgrade log history is needed. To also write a standalone pg_dump tar archive of the logs table, pass --backup-logs-file <absolute path> to upgrade.sh on the host that runs the database (all, db, or remote_db role). Decide this before the upgrade runs: the table cannot be recovered afterwards from anything except a backup. External integrations that query PostgreSQL logs directly will fail after upgrade. Migrate them to kasm_otel.otel_logs.

2. Log file names and JSON fields changed. Each application service now writes one JSON log file: api.log for the API server, which includes the manager, and agent.log for the agent. The api_server.log, api_server_json.log, agent_json.log, manager_api_server.log, manager_api_server_json.log, admin_api_server.log, client_api_server.log, and subscription_api_server.log files are no longer written, and agent.log switches from plain text to JSON. The logging section of api.app.config.yaml and agent.app.config.yaml is no longer read. The KASM_LOG_FILE_* environment variables control the file location and rotation. Update any Filebeat configs, Splunk forwarder paths and field mappings, or custom parsers. For the file mapping, see Log file names consolidated in 1.20.0. The upgrade migrates a syslog handler address from api.app.config.yaml into the OTel Collector, which then forwards records from every service. An address set only in agent.app.config.yaml is not migrated. Re-enable syslog forwarding in the collector configuration after the upgrade if the agent was the only service forwarding to syslog, or if it forwarded to a different host.

Old field	New field
levelname	level
name	logger
lineno	line
asctime	(removed — use timestamp)
processName	(removed)
threadName	(removed)
filename	(removed)
funcName	(removed)
timestamp	timestamp (unchanged)
module	module (unchanged)
message	message (unchanged)

3. Two new containers are added automatically.

Container	Image	Roles
kasm_otel_collector	otel/opentelemetry-collector-contrib:0.160.0	All roles except the dedicated clickhouse role
kasm_clickhouse	clickhouse/clickhouse-server:24.12-alpine	Single-server all installs, or a dedicated clickhouse role node

For offline installations, pre-pull both images before upgrading:

docker pull otel/opentelemetry-collector-contrib:0.160.0
docker pull clickhouse/clickhouse-server:24.12-alpine

Estimated ClickHouse disk requirements at default retention:

Deployment size	Estimated storage
< 50 users	~3 GB
50–500 users	~15 GB
500+ users	~60 GB

4. New network ports (distributed deployments only). Before upgrading remote nodes, allow inbound TCP 4317 on the app/all node from every remote node. Do not open it to the internet. The app role never bundles ClickHouse: in any multi-server deployment the app node needs outbound access to a dedicated clickhouse role node (or an external ClickHouse) on TCP 8443 and 9440, the same as it needs access to the Database Server — see Observability configuration.

5. Console output is now colorized. The console log format uses ANSI color codes by default. To disable, set KASM_LOG_CONSOLE_COLOR=false on each service container.

6. OTel attribute and metric names use updated conventions. Attribute names on logs, traces, and metrics follow OpenTelemetry semantic conventions where a standard applies, and a kasm.<domain>.<attribute> naming scheme for everything Kasm-specific. Any saved ClickHouse query, dashboard, or alert that filters on an attribute name directly needs updating — see the full mapping table in Migrate downstream consumers from PostgreSQL logs to ClickHouse.

Overview

Kasm Workspaces 1.20.0 introduces a built-in observability stack based on OpenTelemetry. Structured logs, distributed traces, and metrics from all core services are collected by a per-node OTel Collector and stored in ClickHouse. This guide covers the breaking changes introduced by the new stack, the upgrade procedure for single-node and distributed deployments, post-upgrade verification, and migration of any downstream consumers that previously queried the PostgreSQL logs table directly.

Prerequisites

• A working Kasm Workspaces installation running version 1.19.x.
• Sufficient disk space for ClickHouse data on whichever node hosts it: the all node in a single-server deployment, or the dedicated clickhouse role node (or external instance) in a multi-server deployment. See the storage estimates in the breaking changes section above.
• For distributed deployments: administrative access to all nodes in the same maintenance window, and the ability to open TCP 4317 on the app/all node firewall from all remote node IP addresses.
• For offline installations: internet access on a staging machine to pre-pull the two new Docker images listed above.
• Confirmation that /opt/kasm/backups/ has space for the pre-upgrade database backup, which the upgrader creates automatically.

Solution approach

This guide progresses through the following phases:

1. Review breaking changes and prepare the environment.
2. Upgrade the app/all node, or complete the single-node upgrade.
3. Upgrade each remote node in a distributed deployment.
4. Verify the observability stack is running and receiving data.
5. Migrate downstream consumers from PostgreSQL logs to ClickHouse.

Detailed steps

Review breaking changes and prepare the environment

1. Read each item in the warning block at the top of this page.
2. For offline installations, pre-pull the two new images on every machine that will run the upgrade:

   docker pull otel/opentelemetry-collector-contrib:0.160.0
   docker pull clickhouse/clickhouse-server:24.12-alpine

3. For distributed deployments, open inbound TCP 4317 on the app/all node firewall from every remote node IP address.
4. Update any Filebeat configs, Splunk forwarder paths and field mappings, or custom log parsers to use the new file names and field names shown in breaking change 2.
5. Generate a support bundle before upgrading if telemetry continuity matters for an active investigation. Services stop during the upgrade, so no telemetry is generated in that window.

Upgrade the app/all node or single-node deployment

Single-node upgrade:

sudo bash upgrade.sh --backup-logs-file /opt/kasm/backups/kasm_logs_backup.tar

--backup-logs-file is optional but strongly recommended: it is the only way to keep the PostgreSQL logs table as a standalone file, and the option is gone once the migration has run. No other flags are required. The upgrader automatically:

1. Recovers the existing OTel token and ClickHouse password from the current install.
2. Preserves a sticky --skip-otel opt-out if one was set on a previous install.
3. Generates fresh credentials if this is the first upgrade from a pre-observability version.
4. Deploys both new containers.

Enabling observability on a previously opted-out install:

If a prior install used --skip-otel, that choice persists through upgrades. Pass --enable-otel to add observability:

sudo bash upgrade.sh --enable-otel

For distributed deployments, enable the app node first. Then retrieve the generated token and enable each remote node:

# On the app node:
sudo bash upgrade.sh --enable-otel

# Retrieve the generated token:
grep -oP '(?<=token: ")[^"]+' /opt/kasm/current/conf/otel/otel-collector-config.yaml

# On each remote node:
sudo bash upgrade.sh --enable-otel \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname>

Distributed — app node upgrade:

sudo bash upgrade.sh -S app -q <db-hostname> -Q <db-password>

After the upgrade completes, retrieve the OTel token. All remote nodes require this value:

grep -oP '(?<=token: ")[^"]+' /opt/kasm/current/conf/otel/otel-collector-config.yaml

Upgrade each remote node

All nodes must share the same OTel bearer token retrieved from the app node. Run the appropriate command for each role.

Agent nodes:

sudo bash upgrade.sh -S agent -p <public-hostname> -m <manager-hostname> -k <manager-token> \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname>

Guac nodes:

sudo bash upgrade.sh -S guac -n <api-hostname> -p <public-hostname> -k <registration-token> \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname>

Proxy nodes:

sudo bash upgrade.sh -S proxy -n <api-hostname> \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname>

DB nodes:

sudo bash upgrade.sh -S db -Q <db-password> \
  --backup-logs-file /opt/kasm/backups/kasm_logs_backup.tar \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname>

Optional — verified TLS on the inter-collector hop:

By default the 4317 hop above is bearer-token authenticated and TLS-encrypted, without certificate verification. If the app/all node uses a CA-signed certificate whose SANs cover the exact hostname passed as --otel-collector-hostname, add --otel-collector-verify-tls (public CA) or --otel-collector-ca-cert <path> (private/corporate CA) to any of the remote-node commands above (agent, guac, proxy — no effect on db):

sudo bash upgrade.sh -S guac -n <api-hostname> -p <public-hostname> -k <registration-token> \
  --otel-token <token-from-app-node> \
  --otel-collector-hostname <app-node-hostname> \
  --otel-collector-ca-cert /path/to/corporate-ca.pem

This choice persists automatically on later upgrades. To turn verification back off on a node that has it recovered forward, pass --otel-collector-no-verify-tls:

sudo bash upgrade.sh -S guac --otel-collector-no-verify-tls

See Configure custom TLS certificates for the full prerequisites, and Diagnose TLS handshake failures on the inter-collector hop if verification fails after enabling it.

Telemetry during the upgrade window:

Remote nodes buffer telemetry in memory during the app/all node upgrade, retrying for approximately 5 minutes. Telemetry emitted beyond that window is dropped. Local log files on every node are unaffected because they are written independently of OTLP export. Upgrade all nodes in the same maintenance window to keep the buffer window as short as possible.

Verify the observability stack

1. Confirm both new containers are running:

   docker ps | grep -E "kasm_otel|kasm_clickhouse"

2. Check the collector health endpoint:

   curl -s http://localhost:13133/ | grep -i healthy

3. Confirm ClickHouse is receiving data (run this on the app/all node):

   docker exec kasm_clickhouse clickhouse-client -u kasm_otel --password '<password>' \
     --query "SELECT count() FROM kasm_otel.otel_logs WHERE Timestamp > now() - INTERVAL 5 MINUTE"

4. For distributed deployments, verify remote collectors are forwarding data:

   docker logs kasm_otel_collector 2>&1 | grep -i "error\|unauthenticated" | tail -20

Migrate downstream consumers from PostgreSQL logs to ClickHouse

Map old logs column queries to kasm_otel.otel_logs using the reference below:

Old logs column	otel_logs expression
ingest_date	Timestamp
message	Body
levelname	SeverityText
host	ResourceAttributes['host.name']
metric_name	LogAttributes['kasm.metric_name']
kasm_user_name	LogAttributes['kasm.kasm.user_name']
kasm_id	LogAttributes['kasm.kasm.id']
server_id	LogAttributes['kasm.server.id']
client_ip	LogAttributes['client.address']
url	LogAttributes['url']
domain	LogAttributes['domain']
category	LogAttributes['category']

Logs forwarded verbatim from components that are not yet OTel-native (the squid adapter, session containers, older agents mid-upgrade) still carry a flat metric_name on the wire, but Kasm remaps the key to kasm.metric_name at ingest, so every query only ever needs to check LogAttributes['kasm.metric_name'].

Example migration query:

-- Old: SELECT message FROM logs WHERE metric_name = 'account.login.failed' LIMIT 10
-- New:
SELECT Body, LogAttributes['kasm.kasm.user_name'] AS kasm_user_name, Timestamp
FROM kasm_otel.otel_logs
WHERE LogAttributes['kasm.metric_name'] = 'kasm.account.login.failed'
ORDER BY Timestamp DESC
LIMIT 10;

Updated OTel attribute and metric names

The table above covers the PostgreSQL-to-ClickHouse column migration. Independently of that migration, several OTel attribute and metric names changed in 1.20.0 to follow OpenTelemetry semantic conventions and a consistent kasm.<domain>.<attribute> naming scheme. Update any saved ClickHouse query, Grafana dashboard, or alert rule that references an old name in the left column. Query otel_logs (LogAttributes) or otel_traces (SpanAttributes) depending on which signal carries the attribute; most of these appear on more than one signal.

Old name	New name	Signal(s)
http.method	http.request.method	traces, metrics
http.target	url.path	traces
http.scheme	url.scheme	traces
http.host	server.address and server.port	traces
http.status_code	http.response.status_code	traces, metrics
request_ip	client.address	logs
user_agent	user_agent.original	logs
kasm_id	kasm.kasm.id	logs, traces
kasm_user_id	kasm.kasm.user_id	logs, traces
kasm_user_name	kasm.kasm.user_name	logs, traces
kasm_image_id	kasm.kasm.image_id	logs
kasm_image_name	kasm.kasm.image_name	logs
kasm_image_friendly_name	kasm.kasm.image_friendly_name	logs, metrics
uptime	kasm.kasm.uptime	logs
zone_name / zone_id	kasm.zone.name / kasm.zone.id	logs, metrics
session.duration_seconds (metric name)	kasm.kasm.duration (unit: seconds)	metrics
server_id / hostname (gauges)	kasm.server.id / kasm.server.hostname	metrics
gpu_id	kasm.server.gpu_id	metrics
realm	kasm.account.realm	metrics
account.* / provision.*	kasm.account.* / kasm.provision.*	metrics, logs
agent.* / manager.* / guac.* / rdp_gateway.* / rdp_https_gateway.*	the same names under kasm.	metrics
metric_name (the log attribute key)	kasm.metric_name	logs
scaling.* / staging.* / performance.* / sessions.* (metric_name values)	the same names under kasm.	logs
session.expired (metric_name value)	kasm.kasm.expired	logs
docker_stats / docker_top	kasm.agent.docker.stats / kasm.agent.docker.top	logs
manager_expiration	kasm.manager.expiration	logs

Fields not listed here — including allow, category, site, and domain — kept their existing names. Those are wire contracts forwarded verbatim from sibling components (the agent, guac, the RD Gateway, the squid adapter, and Kasm session containers), with the sole exception of metric_name, whose key Kasm remaps to kasm.metric_name at ingest so every query needs to check only one spelling regardless of source. Forwarded records are stored under the originating component's ServiceName and carry kasm.log.ingest_path set to legacy_bridge.

Host-resource gauges were renamed and rescaled

The per-component CPU, memory, disk, and GPU gauges changed name and value scale:

Old metric	New metric	Unit	Values
<component>.cpu_percent	kasm.<component>.cpu.utilization	1	ratio [0,1]
<component>.memory_percent	kasm.<component>.memory.utilization	1	ratio [0,1]
<component>.disk_percent	kasm.<component>.disk.utilization	1	ratio [0,1]
<component>.gpu_percent	kasm.<component>.gpu.utilization	1	ratio [0,1]
<component>.gpu_memory_percent	kasm.<component>.gpu.memory.utilization	1	ratio [0,1]
<component>.gpu_temp	kasm.<component>.gpu.temperature	Cel	Celsius

<component> is one of agent, manager, guac, rdp_gateway, rdp_https_gateway.

warning

Utilization values are now ratios rather than percentages. If you query otel_metrics_gauge directly — in Grafana, a custom dashboard, or an export job — multiply by 100 to keep producing percentages. Kasm's own admin reports and health alerting are unaffected.

For the full field-by-field reference, see Observability metrics reference and Reporting and logging.

Rollback

If the upgrade must be reversed:

1. Stop the new version:

   /opt/kasm/bin/stop

2. Restore the database backup:

   /opt/kasm/<previous-version>/bin/utils/db_restore -a \
     -f /opt/kasm/backups/kasm_db_backup.tar \
     -p /opt/kasm/<previous-version>

3. Update the symlink:

   ln -sfn /opt/kasm/<previous-version> /opt/kasm/current

4. Start the previous version:

   /opt/kasm/bin/start

Restoring the database backup recreates the PostgreSQL logs table with its pre-upgrade contents. ClickHouse data written by the new version is not affected by the rollback and can be removed manually if needed.

Common troubleshooting steps

• New containers do not start. Run docker logs kasm_otel_collector and docker logs kasm_clickhouse to check startup errors. Confirm that sufficient disk space is available on the app/all node.

• Remote nodes log "unauthenticated" errors. The OTel token on the remote node does not match the token on the app node. Retrieve the correct token from the app node with grep -oP '(?<=token: ")[^"]+' /opt/kasm/current/conf/otel/otel-collector-config.yaml, then re-run the remote upgrade with --otel-token <correct-token>.

• ClickHouse receives no data after upgrade. Confirm inbound TCP 4317 is open on the app/all node firewall. Check the collector health endpoint: curl -s http://localhost:13133/.

• TLS handshake or certificate errors after passing --otel-collector-verify-tls/--otel-collector-ca-cert. See Diagnose TLS handshake failures on the inter-collector hop for diagnosis, and --otel-collector-no-verify-tls for a durable rollback.

• SIEM forwarder drops log fields. Verify field name mappings against the JSON log field renames table in the breaking changes section. Fields removed in 1.20.0 (asctime, processName, threadName, filename, funcName) do not appear in new log output.

• Console output contains unexpected characters. The console log format now uses ANSI color codes. Set KASM_LOG_CONSOLE_COLOR=false on each service container to disable color output.

• Rollback leaves ClickHouse data in place. Rolling back restores only the PostgreSQL database. ClickHouse data from the upgraded version remains on disk but is not used by the rolled-back install. Remove it manually once the rollback is confirmed stable.