Observability configuration reference

This page is the complete configuration reference for the Kasm Workspaces observability stack. For a conceptual overview of how the components fit together, see Kasm Workspaces observability.

Install and upgrade flags

Pass these flags to install.sh or upgrade.sh to control observability behavior. Flags marked as sticky persist across upgrades unless overridden.

Flag	Default	Description
--skip-otel	Observability enabled	Disable observability. Sticky opt-out: subsequent upgrades skip observability unless --enable-otel is passed.
--enable-otel	(see above)	Re-enable observability on a node where --skip-otel was previously set.
--otel-token <token>	Auto-generated, 32 chars	Inter-collector authentication token used between remote and central collectors.
--otel-collector-hostname <host>	(app node hostname)	Hostname of the central collector. Set on remote nodes to point their collectors at the correct central collector.
--otel-collector-verify-tls	off (encrypt-only)	Verify the central collector's certificate on the inter-collector hop (port 4317) instead of skipping verification. Set on remote nodes (agent, guac, proxy, db); no effect on all/app/clickhouse (no otlp/forward exporter). Requires the app/all node to be installed with a CA-signed certificate whose SANs cover the hostname passed as --otel-collector-hostname. Verifies against the collector image's system trust store — sufficient for a publicly-trusted CA. Sticky: persists across upgrades. See Enable verified TLS on the inter-collector hop.
--otel-collector-ca-cert <path>	unset	Implies --otel-collector-verify-tls. Copies <path> to certs/otel_ca.crt on the remote node and configures the exporter to trust it — the recommended option for a private or corporate CA, and works for a public CA too. Same role scope and persistence as --otel-collector-verify-tls.
--otel-collector-no-verify-tls (upgrade.sh only)	n/a	Explicitly clears a --otel-collector-verify-tls/--otel-collector-ca-cert choice that a prior upgrade recovered forward, restoring encrypt-only TLS on that node. Not available on install.sh — a fresh install with neither flag already defaults to encrypt-only.
--otel-collector-endpoint <url>	(local collector)	Bring-your-own collector endpoint. Services emit OTLP to this URL instead of the bundled collector.
--clickhouse-endpoint <url>	all: bundled ClickHouse. app: none — required unless --no-clickhouse is set	Native DSN for a dedicated (--role clickhouse) or external ClickHouse instance. A full DSN is used verbatim; a bare host:port is wrapped as tcp://host:port?dial_timeout=10s. If the DSN contains secure=true without an explicit skip_verify parameter, install.sh appends skip_verify=true (no certificate authority is distributed between nodes, so the collector does not verify the ClickHouse Server's certificate; the connection is still encrypted). This matches the Database Server's default TLS posture (sslmode=require): encrypted in transit, without certificate verification. Pass skip_verify=false explicitly if you manage your own CA-signed certificate for the ClickHouse instance and want the collector to verify it. Requires --clickhouse-http-endpoint. Set on app/all roles; on app, if omitted (and --no-clickhouse is not set), install.sh prompts interactively for the ClickHouse Server hostname, exactly like it prompts for the database hostname.
--clickhouse-http-endpoint <url>	all: bundled ClickHouse. app: none — required unless --no-clickhouse is set	HTTPS URL of the same dedicated or external ClickHouse instance, used by the Kasm UI for report queries. Required together with --clickhouse-endpoint. Valid alone only when --otel-collector-endpoint is also set. Set on app/all roles.
--clickhouse-username <user>	kasm_otel	Username for the writer user, used by the OTel Collector's exporter to write telemetry to the dedicated or external ClickHouse instance over the native TCP interface. On the clickhouse role, sets the writer user to create. On app/all, this is the write-path credential used to connect to the dedicated or external ClickHouse instance.
--clickhouse-password <pass>	Auto-generated, 20 chars	Password for the writer user. On the clickhouse role, sets the ClickHouse server password for the writer user (kasm_otel by default). On the all role, sets the password for the bundled ClickHouse writer user. On the app role, the connect password used to reach the dedicated or external ClickHouse instance — there is no bundled ClickHouse to connect to on this role.
--clickhouse-reader-username <user>	kasm_otel_ro	Username for the reader user, used by kasm_api and kasm_manager to read telemetry for the built-in reports and log viewer over the HTTPS interface. On the clickhouse role, sets the reader user to create. On app/all, this is the read-path credential used to connect to the dedicated or external ClickHouse instance.
--clickhouse-reader-password <pass>	Auto-generated, 20 chars	Password for the reader user. On the clickhouse role, sets the ClickHouse server password for the reader user (kasm_otel_ro by default). On the all role, sets the password for the bundled ClickHouse reader user. On the app role, the connect password used to reach the dedicated or external ClickHouse instance for report and log-viewer queries. Omitting this on app/all when pointing at a dedicated or external instance causes UI reports and the log viewer to fail authentication.
--no-clickhouse	all: bundled ClickHouse enabled. app: ClickHouse required	On the all role, omits the bundled ClickHouse container. On the app role, runs the node without any ClickHouse — the collector starts in debug exporter mode (see Debug exporter mode) instead of connecting to a dedicated or external instance. Mutually exclusive with --clickhouse-endpoint and --clickhouse-http-endpoint.
--otel-forward-endpoint <url>	(none)	Forward all signals in parallel to this OTLP-compatible endpoint (e.g., Grafana Cloud or Splunk).
--otel-forward-token <token>	(none)	Bearer token sent with forwarded signals. Required when --otel-forward-endpoint requires authentication.
--otel-trace-sample-percentage <n>	10	Percentage of traces retained by probabilistic tail sampling after error and latency policies are applied.
--otel-logs-retention-days <n>	30	Retention period in days for application log records in ClickHouse.
--otel-audit-logs-retention-days <n>	365	Retention period in days for audit-classified log records in ClickHouse.
--otel-traces-retention-days <n>	7	Retention period in days for trace spans in ClickHouse.
--otel-metrics-retention-days <n>	90	Retention period in days for metric data points in ClickHouse.
--otel-db-monitor-password <pass>	Auto-generated on db/all; prompted on app if omitted	Password for the kasm_otel_monitor PostgreSQL role used by the collector's postgresql metrics receiver. Set automatically when the role is created, on the db/all role. On the app role, whose database is external, pass the same password the db/all role generated, or install.sh prompts for it interactively. See PostgreSQL monitoring role.
--backup-file-name <path> (upgrade.sh only)	/opt/kasm/backups/kasm_db_backup.tar	Absolute path of the pre-upgrade database backup tar. The rollback restore step reads the same path. The path must not be an existing directory, and its parent directory must exist or be creatable. The file is owned by 70:70 with mode 640, like the default backup.
--backup-logs-file <path> (upgrade.sh only)	not created	Absolute path of an additional tar containing only the PostgreSQL logs table, taken before the schema migration drops it. Valid for the all, db, and remote_db roles. Skipped with a message when the previous database has no logs table.
-N, --skip-connection-test	checks run	Skips the pre-upgrade connectivity checks described in Installer connectivity checks, along with the database and API health checks. Accepted by install.sh and upgrade.sh.

Debug exporter mode

When --no-clickhouse is set and no dedicated or external ClickHouse is configured (no --clickhouse-endpoint or --otel-collector-endpoint), the collector starts in debug exporter mode. In this mode, the collector logs all received signals to its own standard output and does not persist any telemetry data. All Kasm business functionality remains available.

Installer connectivity checks

install.sh and upgrade.sh run the same connectivity checks before changing the host. On upgrade, the checks run before the running install is stopped.

Role	Check	On failure
agent, guac, proxy, db	TCP reachability of the central collector on port 4317 at --otel-collector-hostname (default: the API server hostname). Skipped when --otel-collector-endpoint is set or when no collector hostname is known.	Warning only. Remote roles are often installed before the app node exists.
app, or all, with a dedicated or external ClickHouse	TCP reachability of the native and HTTP endpoints, then an authenticated SELECT 1 over the HTTP interface as the reader user and as the writer user.	Fatal. The install or upgrade stops.

The ClickHouse check does not run with --skip-otel or --no-clickhouse, and it has nothing to test when no dedicated or external ClickHouse endpoint is configured. An all node with the bundled ClickHouse is not checked. When only --clickhouse-endpoint is set, the HTTP URL is assumed: https://<host>:8443 when the DSN carries secure=true, otherwise http://<host>:8123. A failure to reach an assumed URL is a warning.

Credentials reach the check through the environment and reach curl on standard input, never on a command line, and are never printed.

-N, --skip-connection-test skips all of these checks along with the database and API health checks.

Dedicated ClickHouse role

Install --role clickhouse to run ClickHouse (kasm_clickhouse) on its own dedicated host. The app role never bundles ClickHouse — in a multi-server deployment it must connect to this dedicated role or to an external ClickHouse instance, exactly like the Database Server role. ClickHouse is bundled on the node only in a single-server (all) installation.

• The clickhouse role runs no OTel Collector and does not forward telemetry — it is a storage endpoint only.
• ClickHouse listens on TLS-only ports published to the network: 8443 (HTTPS) and 9440 (secure native TCP). There is no plaintext or Docker-internal-only mode for this role.
• The clickhouse role provisions two users: a writer (kasm_otel by default, set with --clickhouse-username/--clickhouse-password) and a reader (kasm_otel_ro by default, set with --clickhouse-reader-username/--clickhouse-reader-password). install.sh prints both credential pairs, along with the exact flags to pass to the app node, after provisioning completes.
• Point the app/all node at the dedicated instance using --clickhouse-endpoint, --clickhouse-http-endpoint, and both credential pairs — the writer pair for the OTel Collector's write path, and the reader pair for the kasm_api/kasm_manager read path — for example:

install.sh --role app ... \
  --clickhouse-endpoint "tcp://ch-host:9440?secure=true&skip_verify=true" \
  --clickhouse-http-endpoint "https://ch-host:8443" \
  --clickhouse-username kasm_otel \
  --clickhouse-password [CLICKHOUSE_PASSWORD] \
  --clickhouse-reader-username kasm_otel_ro \
  --clickhouse-reader-password [CLICKHOUSE_READER_PASSWORD]

note

Omitting the reader credential pair on app/all causes the Kasm UI's built-in reports and log viewer to fail authentication against the dedicated or external ClickHouse instance, even though telemetry ingestion continues to work normally.

• Retention/TTL of telemetry is managed on whichever node hosts ClickHouse — the clickhouse role, or all for single-server deployments. When the app/all node points at a dedicated or external ClickHouse, retention flags passed to that node have no effect.
• The app/all node must reach the clickhouse node on TCP 8443 (HTTPS reads by kasm_api/kasm_manager) and 9440 (secure native writes by the collector). See Ports.

ClickHouse writer and reader credentials

Kasm Workspaces provisions two ClickHouse users, each scoped to a single direction of data flow.

User	Default username	Set with	Used by	Interface	Required grants
Writer	kasm_otel	--clickhouse-username / --clickhouse-password	OTel Collector exporter (write path)	Native TCP (9440/9000)	SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW
Reader	kasm_otel_ro	--clickhouse-reader-username / --clickhouse-reader-password	kasm_api/kasm_manager (read path)	HTTPS (8443/8123)	SELECT

This split follows the principle of least privilege: a compromised writer credential can append data but cannot rewrite or delete already-retained records, and the read path never needs write access.

For a bundled (all) installation, both users and passwords are generated automatically and no operator action is required.

For a customer-managed external or SaaS ClickHouse instance, create both users manually and grant each only what it needs, for example:

CREATE USER kasm_otel IDENTIFIED WITH sha256_password BY '[CLICKHOUSE_PASSWORD]';
GRANT SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW ON kasm_otel.* TO kasm_otel;

CREATE USER kasm_otel_ro IDENTIFIED WITH sha256_password BY '[CLICKHOUSE_READER_PASSWORD]';
GRANT SELECT ON kasm_otel.* TO kasm_otel_ro;

The writer requires CREATE DATABASE because the collector's ClickHouse exporter issues CREATE DATABASE IF NOT EXISTS on every start, in addition to CREATE TABLE and CREATE VIEW for schema migrations. The reader requires only SELECT, since kasm_api and kasm_manager never write to ClickHouse.

PostgreSQL monitoring role (kasm_otel_monitor)

The OTel Collector's postgresql metrics receiver (see PostgreSQL database metrics) authenticates as a dedicated kasm_otel_monitor role, granted the built-in PostgreSQL pg_monitor role — read-only visibility into database activity statistics, with no access to application data.

note

Unlike the ClickHouse writer and reader passwords, the kasm_otel_monitor password is not something you choose at every install. It is generated automatically on the db/all role, where the role is created, and recovered automatically on subsequent upgrades. The one case requiring manual input is the app role: since its database is external, install.sh prompts for the existing kasm_otel_monitor password, or accepts it directly with --otel-db-monitor-password, the same pattern used for the ClickHouse password prompt on that role.

PostgreSQL logs require no separate credential — the collector reads the mounted log file directly. postgresql.conf writes both the existing stderr text log (for operators reading the file directly) and a jsonlog structured log (for the collector to parse), a small increase in log I/O that mirrors the trade-off nginx's access log makes with its own JSON log format.

For a customer-managed external PostgreSQL instance, create the role manually and grant it pg_monitor, for example:

CREATE ROLE kasm_otel_monitor WITH LOGIN PASSWORD '[OTEL_DB_MONITOR_PASSWORD]';
GRANT pg_monitor TO kasm_otel_monitor;

Environment variables

Environment variables configure individual service behavior and override default SDK settings. They are set in the service container environment; contact your Kasm support channel for procedures to apply them in production.

Standard OTel variables: SDK kill-switches

These variables disable OTLP export without affecting console or file output.

Variable	Default	Description
OTEL_SDK_DISABLED	false	Set true to disable all OTLP export. Console and file channels are unaffected.
OTEL_LOGS_EXPORTER	otlp	Set none to disable OTLP log export only.
OTEL_TRACES_EXPORTER	otlp	Set none to disable trace export only.
OTEL_METRICS_EXPORTER	otlp	Set none to disable metrics export only.

Standard OTel variables: transport and identity

Variable	Default	Description
OTEL_SERVICE_NAME	(per-service value)	Service identity label attached to all signals from this process.
OTEL_RESOURCE_ATTRIBUTES	(none)	Additional resource attributes, e.g. service.instance.id=node-01.
OTEL_EXPORTER_OTLP_ENDPOINT	http://kasm_otel_collector:4318	Collector HTTP endpoint for all OTLP signals.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf	Transport protocol for OTLP export.
OTEL_EXPORTER_OTLP_TIMEOUT	10	Export timeout in seconds.
OTEL_PYTHON_LOG_CORRELATION	true	Inject trace_id and span_id into log records.
OTEL_PYTHON_EXCLUDED_URLS	socket.io/*,healthcheck	URL path patterns excluded from trace instrumentation.

Kasm-specific variables

Variable	Default	Description
KASM_LOG_LEVEL	INFO	Log level for OTLP export and file output.
KASM_LOG_CONSOLE_LEVEL	INFO	Log level for console (stdout) output. Can differ from KASM_LOG_LEVEL.
KASM_LOG_CONSOLE_COLOR	true	Colored console output. Set false for plain text (useful for log aggregators).
KASM_LOG_FILE_ENABLED	true	Enable file logging. Set false in Kubernetes where a log collector manages files.
KASM_LOG_FILE_PATH	/opt/kasm/current/log/api.log on the API server, /opt/kasm/current/log/agent.log on the agent	Active log file. The manager runs inside the API server process and writes to api.log.
KASM_LOG_FILE_WHEN	midnight	Time-based rotation unit: s (seconds), m (minutes), h (hours), d (days), midnight, or w0 to w6 (weekly, Monday to Sunday), case-insensitive. Any other value fails startup with a validation error. midnight is midnight UTC. The rotated file name carries a timestamp at the unit's granularity, for example api.log.2026-10-08 for midnight and api.log.2026-10-08_14 for h.
KASM_LOG_FILE_INTERVAL	1	Number of KASM_LOG_FILE_WHEN units between time-based rotations. h with 6 rotates every six hours. It also applies to midnight and w0 to w6, so midnight with 6 rotates every six days. Leave it at 1 for daily or weekly rotation. Must be at least 1.
KASM_LOG_FILE_MAX_BYTES	10485760	File size in bytes that also triggers a rotation. Default is 10 MB. 0 disables the size trigger. For s, m, h, and d, a size rotation also restarts the time schedule. A size rotation in a time window that already has a backup renames that backup with a counter (.1, .2, and so on) instead of overwriting it.
KASM_LOG_FILE_BACKUP_COUNT	20	Rotated files kept per service, counted across time-based and size-based rotations together, including counter-suffixed files. The oldest files are deleted at each rotation. 0 keeps every rotated file.
KASM_LOG_FILE_EXTRA_FIELDS	(empty)	Comma-separated OTel log attribute names to add to each JSON log-file line, for example kasm.metric_name,kasm.kasm.id,kasm.zone.name. Dots become underscores in the file (kasm_metric_name). Names a service never sets are ignored, so one list works for every service. Empty keeps the six base fields. Takes effect on the next process start.
KASM_CLICKHOUSE_ENDPOINT	http://kasm_clickhouse:8123	HTTP(S) URL of the ClickHouse instance used by the Kasm UI for report queries. Set from --clickhouse-http-endpoint when a dedicated or external ClickHouse is configured.
KASM_CLICKHOUSE_USER	kasm_otel_ro	ClickHouse reader username used by kasm_api/kasm_manager for report queries. Sourced from --clickhouse-reader-username. Distinct from the writer user used by the OTel Collector.
KASM_CLICKHOUSE_PASSWORD	(generated at install)	ClickHouse reader password used for report queries. Sourced from --clickhouse-reader-password.

Collector variables

These variables are set in the kasm_otel_collector container environment.

Variable	Default	Description
GRAFANA_API_KEY	(none)	Base64-encoded instanceId:apiKey for Grafana Cloud forwarding.
GRAFANA_OTLP_ENDPOINT	https://otlp-gateway-prod-us-east-3.grafana.net/otlp	Grafana Cloud OTLP endpoint. Override when using a different Grafana region.
OTEL_BEARER_TOKEN	(generated at install)	Bearer token for authenticating remote collectors to the central collector.
OTEL_DB_MONITOR_PASSWORD	(generated at install; see --otel-db-monitor-password)	Password the postgresql metrics receiver uses to authenticate as kasm_otel_monitor.

Ports

Port	Protocol	Scope	Description
4317	TCP gRPC	Inbound on app / all nodes	Remote collectors forward signals to the central collector.
4318	TCP HTTP	Docker network only	Services emit OTLP/HTTP to the local collector. Not exposed to the host network.
4316	TCP HTTP	Docker network only	nginx browser telemetry proxy (internal routing). Not exposed to the host network.
8090	TCP HTTP	Docker network only	nginx stub_status endpoint scraped by the collector for nginx metrics.
13133	TCP HTTP	Host-local	Collector health check endpoint. Accessible on the host at localhost:13133.
8443	TCP HTTPS (TLS)	Inbound on clickhouse role nodes	Secure ClickHouse HTTP interface, published to the network. Read by kasm_api/kasm_manager for UI reports.
9440	TCP native (TLS)	Inbound on clickhouse role nodes	Secure ClickHouse native interface, published to the network. Written to by the OTel Collector.

note

Port 4317 is required between nodes in any distributed deployment. Ports 8443 and 9440 do not apply to a single-server all installation, where ClickHouse is bundled and the interfaces remain Docker-internal only. In any multi-server deployment, the app node has no bundled ClickHouse and requires these ports to reach the dedicated clickhouse role node or an external ClickHouse instance. Do not expose any of these ports to the internet.

The postgresql metrics receiver on the db/all role's collector connects to the Database Server on the existing database port (5432 by default, set with --db-port) — it opens no additional port.

Container and image inventory

Container	Image	Deployed on
kasm_otel_collector	otel/opentelemetry-collector-contrib:0.160.0	All node types except the dedicated clickhouse role
kasm_clickhouse	clickhouse/clickhouse-server:24.12-alpine	Single-server all installs, or a dedicated clickhouse role node

To pre-pull images for offline or air-gapped installs, run the following commands.

docker pull otel/opentelemetry-collector-contrib:0.160.0
docker pull clickhouse/clickhouse-server:24.12-alpine

Config file locations

Path	Description
/opt/kasm/current/conf/otel/otel-collector-config.yaml	Deployed collector configuration. The installer writes either the local-node or remote-node variant here.
/opt/kasm/current/conf/nginx/services.d/stub_status.conf	nginx stub_status endpoint configuration (port 8090). Deployed only when observability is enabled.
/opt/kasm/current/log/postgres/	PostgreSQL log directory, mounted read-only into the collector at /var/log/postgres/external on the db/all role for the filelog/postgres receiver.

ClickHouse schema

All telemetry data is stored in the kasm_otel database. The tables follow the standard OpenTelemetry ClickHouse exporter schema.

Table	Contents
otel_logs	All log records. Kasm-specific fields are in the LogAttributes map column.
otel_traces	All trace spans.
otel_traces_trace_id_ts	Trace ID to time range index used for trace lookup. Populated by the otel_traces_trace_id_ts_mv materialized view. Its retention TTL applies to the Start column.
otel_metrics_sum	Counter metrics (monotonic and non-monotonic sums).
otel_metrics_gauge	Gauge metrics.
otel_metrics_histogram	Histogram metrics.
otel_metrics_exponential_histogram	Exponential-histogram metrics.
otel_metrics_summary	Summary metrics.

Retention

Default retention periods

Signal	Default retention
Logs	30 days
Audit logs	365 days
Traces	7 days
Metrics	90 days

Changing retention on an existing install

Re-run the upgrade script with the relevant retention flags to update ClickHouse TTL policies.

sudo bash upgrade.sh \
  --otel-logs-retention-days 60 \
  --otel-audit-logs-retention-days 730 \
  --otel-traces-retention-days 14 \
  --otel-metrics-retention-days 180

The upgrade applies ALTER TABLE ... MODIFY TTL to otel_logs, otel_traces, otel_traces_trace_id_ts, and the five otel_metrics_* tables, then updates the collector configuration. The statements run as the ClickHouse container's loopback-only default user, so no writer or reader credentials are involved. Retention flags passed to an app node that points at a dedicated or external ClickHouse apply nothing.

Outcome	Output	Exit status
Every statement succeeds	ClickHouse retention tiers applied.	Zero
One or more statements fail	One ERROR: <table>: <ClickHouse error text> line per failed table, then a summary line listing the failed tables	Non-zero
Run reports No upgrade necessary	The same success or failure output as above. Retention flags are applied even when nothing else changes.	As above

note

Reducing retention on an existing install does not immediately free disk space. ClickHouse drops expired data parts on its next TTL merge cycle, which typically occurs within a few hours.

Audit log classification

A log record is classified as an audit record (and is subject to the longer audit retention period) when its kasm.retention_class attribute is set to audit. The retention rule keys on that attribute directly, so the same attribute is what any log backend or viewer filters on. Kasm services stamp it on the following records:

• API login success and logout.
• Authentication failures for local and LDAP credentials, and password logins refused on SSO accounts.
• Authentication success and failure on the Connection Proxy (Guac) and on both RDP gateways.
• Two-factor enrollment and verification failures.
• Account lockout and disabled-account refusals.
• API key rejections: invalid, expired, read-only misuse, or insufficient authorization.
• Permission denied (HTTP 403) responses on the legacy and native API paths.
• Administrative system and group settings changes. The record carries setting names only, never values.

Sampling

The central collector applies tail-based sampling to distributed traces. Sampling decisions are made after the collector observes a complete (or near-complete) trace. The following policies are evaluated in order.

1. Error policy: keep all traces that contain at least one span with ERROR status.
2. Latency policy: keep all traces whose observed duration is 1000 ms or greater.
3. Probabilistic policy: keep the configured percentage of remaining traces (default 10%, set with --otel-trace-sample-percentage).

The decision window is 30 seconds. Traces that take longer than 30 seconds to complete (such as slow virtual machine provisioning operations) may be evaluated on partial span data.

note

The 30-second decision window is not user-configurable in this release.

ClickHouse exporter sending queue

Each ClickHouse exporter (logs, traces, and metrics have separate exporters) is configured with a local in-memory sending queue. This queue provides buffering against transient ClickHouse unavailability.

Setting	Value
Export consumers	4
Queue size	1,000 slots
Retry initial delay	5 s
Retry maximum delay	30 s
Retry maximum elapsed	300 s

With a 300-second (5-minute) maximum elapsed retry time, the queue can absorb short ClickHouse outages without data loss. Records that remain undelivered after 300 seconds are dropped.