Skip to main content
Version: Developer

Troubleshoot observability

Overview​

When the observability stack is not delivering data as expected, symptoms typically fall into one of several categories: no data reaching ClickHouse, the collector not receiving signals from a service, authentication failures on remote nodes, missing log files, or nginx proxy or PostgreSQL database telemetry gaps. This guide provides a per-symptom diagnostic sequence that isolates the failure layer and identifies a resolution. Work through the section that best matches your symptom first.

Prerequisites​

  • Root access to the affected Kasm Workspaces host or hosts.
  • The observability stack installed and not skipped with --skip-otel.
  • Familiarity with basic Docker commands.

Solution approach​

This guide progresses through the following phases:

  1. Diagnose missing telemetry data in ClickHouse.
  2. Diagnose collector ingestion failures, remote-node token mismatches, and TLS handshake failures on the inter-collector hop.
  3. Diagnose missing log files and nginx- or PostgreSQL-specific telemetry gaps.
  4. Diagnose ClickHouse availability issues.

Detailed steps​

Diagnose missing telemetry data in ClickHouse​

When ClickHouse contains no records or records stop arriving, follow these steps.

  1. Verify the service container has the correct endpoint configured:

    docker inspect kasm_api | grep -i otel

  2. Check the collector logs for export errors:

    docker logs kasm_otel_collector

  3. Verify that ClickHouse is receiving data:

    docker exec kasm_clickhouse clickhouse-client -u kasm_otel --password '<password>' \
      --query "SELECT count() FROM kasm_otel.otel_logs"

    A count of zero after services have been running for several minutes indicates an export failure at the collector or ClickHouse layer.

Diagnose collector ingestion failures​

When services are running but the collector is not receiving data from them, follow these steps.

  1. Confirm that the service container can reach the collector on the Docker network:

    docker network inspect kasm_default_network | grep otel

  2. Check the collector logs for pipeline or connection errors:

    docker logs kasm_otel_collector

  3. Verify that ClickHouse is running and accepting connections from the collector:

    docker logs kasm_otel_collector 2>&1 | grep -i clickhouse

  4. Confirm that the ClickHouse credentials in the collector configuration match the ClickHouse instance:

    grep -i password /opt/kasm/current/conf/otel/otel-collector-config.yaml

Resolve remote-collector token mismatches​

When remote nodes cannot authenticate to the central collector, the central collector drops their telemetry silently. This results in missing data for those nodes in ClickHouse with no obvious error visible in the service logs on the remote node.

  1. Check the remote collector logs for authentication errors:

    docker logs kasm_otel_collector 2>&1 | grep -i "auth\|401\|forbidden\|unauthenticated"

  2. Compare the OTel bearer token on the remote node with the value on the app or all node:

    # On the remote node:
    grep OTEL_BEARER_TOKEN /opt/kasm/current/conf/otel/otel-collector-config.yaml

    # On the app or all node:
    grep OTEL_BEARER_TOKEN /opt/kasm/current/conf/otel/otel-collector-config.yaml

    Both values must match. If they differ, re-run the upgrade script on the remote node and pass the correct token with --otel-token <correct-token>.

Diagnose TLS handshake failures on the inter-collector hop (4317)​

This applies only if you passed --otel-collector-verify-tls and/or --otel-collector-ca-cert on a remote node (see Enable verified TLS on the inter-collector hop) to turn on certificate verification for the 4317 hop. By default this hop is encrypt-only and none of the following applies.

  1. Check the remote node's collector logs for the specific TLS error:

    docker logs kasm_otel_collector 2>&1 | grep -i "x509\|tls\|certificate"

    Match the error against the likely cause:

    • x509: certificate is valid for X, not Y — the app/all node's certificate does not have a Subject Alternative Name (SAN) covering the hostname (or IP) this remote node uses in --otel-collector-hostname. Reissue the certificate with the correct SAN, or repoint this node at a hostname the certificate already covers.
    • x509: certificate signed by unknown authority — the CA that signed the app/all node's certificate is not trusted by the remote collector, and no --otel-collector-ca-cert was passed (or it points at the wrong file). Re-run the install or upgrade with the correct --otel-collector-ca-cert <path>.
    • x509: certificate has expired or is not yet valid — the app/all node's certificate was rotated or has expired. Confirm which certificate the node currently presents (see step 3) and reissue if needed.
    • Connection resets or timeouts with no x509 detail — confirm TCP 4317 is reachable at all (see Network prerequisite: TCP 4317) before assuming it is a certificate problem.

  2. On the app/all node, check for a matching-side symptom:

    docker logs kasm_otel_collector 2>&1 | grep -i "tls\|handshake"

    If the app/all node shows nothing at all for the relevant time window, the connection is not reaching it — check firewall and security-group rules rather than the certificate.

  3. Confirm which certificate the app/all node is actually presenting, from the remote node:

    openssl s_client -connect kasm-app.example.com:4317 -showcerts </dev/null 2>/dev/null \
      | openssl x509 -noout -subject -issuer -dates

    Compare the subject/SANs against the hostname you configured, and the validity dates against the current date.

Temporary rollback. To restore the hop to working order immediately while you fix the certificate, hand-edit the rendered configuration on the affected remote node and restart the collector:

sudo sed -i 's/insecure_skip_verify: false/insecure_skip_verify: true/' \
  /opt/kasm/current/conf/otel/otel-collector-config.yaml
docker restart kasm_otel_collector

This restores encrypt-only TLS (still bearer-token authenticated) and telemetry resumes flowing. It is a stopgap only: the next upgrade.sh run recovers this node's verify-tls choice from whatever the rendered configuration says at that time, so it will not silently re-enable verification behind your back — but it also does not durably record that you want verification off going forward.

Durable rollback. Because upgrade.sh recovers a node's existing verify-tls/CA-cert choice on every upgrade, re-running upgrade.sh/install.sh without the flags is not enough to permanently disable verification once it has been recovered forward at least once. Pass --otel-collector-no-verify-tls to explicitly clear it:

sudo bash upgrade.sh --role guac --otel-collector-no-verify-tls

Diagnose a missing log file​

When a service log file does not appear under /opt/kasm/current/log/, follow these steps.

  1. Verify that KASM_LOG_FILE_ENABLED is set to true:

    docker inspect kasm_api | grep KASM_LOG_FILE

  2. Confirm that the log directory exists and is writable:

    ls -la /opt/kasm/current/log/

  3. Check the service logs for file-creation errors:

    docker logs kasm_api 2>&1 | grep -i "log file\|could not open"

Diagnose nginx proxy telemetry gaps​

When ClickHouse contains no records with ResourceAttributes['service.name'] = 'kasm-proxy', the issue is usually a missing volume mount, a group permission gap, or a missing stub_status configuration.

Step 1. Verify the nginx log volume is mounted in the collector:

docker inspect kasm_otel_collector | grep -A5 nginx

The output must show /opt/kasm/current/log/nginx mapped to /var/log/nginx/external. If the mapping is absent, the collector was deployed without the nginx log volume. Re-run the installer or upgrade script to correct this.

Step 2. Verify the collector runs with supplemental group adm (gid 4):

docker inspect kasm_otel_collector | grep -i group

The output must include "GroupAdd": ["4"]. If this entry is absent, the collector cannot read nginx log files because nginx log files are owned by the adm group.

Step 3. Confirm the nginx log files exist and contain data:

ls -la /opt/kasm/current/log/nginx/

Step 4. Check the collector logs for filelog receiver errors:

docker logs kasm_otel_collector 2>&1 | grep -i "filelog\|nginx"

For nginx metrics specifically, verify that the nginx stub_status endpoint is reachable from the collector:

docker exec kasm_otel_collector wget -qO- http://kasm_proxy:8090/stub_status

Expected output begins with Active connections:. If this command fails, check whether stub_status.conf is deployed:

ls /opt/kasm/current/conf/nginx/services.d/stub_status.conf

This file is deployed only when the observability stack is enabled. If it is absent, the observability stack was skipped during installation. Re-run the installer or upgrade script with --enable-otel.

Also confirm that both containers share the same Docker network:

docker network inspect kasm_default_network | grep -E "kasm_proxy|kasm_otel"

Diagnose PostgreSQL database telemetry gaps​

When ClickHouse contains no records with ResourceAttributes['service.name'] = 'kasm-db', the issue is usually a missing volume mount, a missing kasm_otel_monitor role or password mismatch, or a collector that never started on the db role.

Step 1. Verify the PostgreSQL log volume is mounted in the collector:

docker inspect kasm_otel_collector | grep -A5 postgres

The output must show /opt/kasm/current/log/postgres mapped to /var/log/postgres/external. If the mapping is absent, the collector was deployed without the PostgreSQL log volume. Re-run the installer or upgrade script to correct this.

Step 2. Confirm the PostgreSQL log files exist and are readable:

ls -la /opt/kasm/current/log/postgres/

Look for postgresql-*.json files. postgresql.conf must include jsonlog in log_destination and log_file_mode = 0644 — if the files are present but unreadable by the collector, verify these settings were not overridden by a custom postgresql.conf.

Step 3. Check the collector logs for filelog receiver errors:

docker logs kasm_otel_collector 2>&1 | grep -i "filelog\|postgres"

For PostgreSQL metrics specifically, authentication failures against the postgresql receiver appear in the same collector log stream:

docker logs kasm_otel_collector 2>&1 | grep -i "postgresql receiver\|password authentication failed"

A password authentication failure means the OTEL_DB_MONITOR_PASSWORD configured on the collector does not match the kasm_otel_monitor role's actual password on the Database Server. This is most common on the app role after a distributed install where --otel-db-monitor-password was omitted or entered incorrectly — re-run install.sh/upgrade.sh on the app node with the correct password recovered from the db node. See PostgreSQL monitoring role.

Also confirm that the collector can reach the Database Server on its configured port:

docker exec kasm_otel_collector nc -zv <database-hostname> <database-port>

Diagnose ClickHouse availability​

When ClickHouse is unreachable, the Kasm platform continues operating normally. Telemetry-derived reports and dashboards return empty results with a warning. This is the expected degraded behavior; no data is lost from running services.

  1. Check the ClickHouse health endpoint:

    curl -s http://localhost:13133/ | grep -i healthy

  2. Confirm the container is running:

    docker ps | grep kasm_clickhouse

  3. If the container has stopped, inspect its logs for the failure reason:

    docker logs kasm_clickhouse

For collector health check endpoint details and support bundle generation, see Support bundle.

Common troubleshooting steps​

  • Symptom does not match any section above. Run docker logs kasm_otel_collector and search for error or warn. The collector logs the component name alongside each error, which identifies the failing pipeline stage.
  • ClickHouse container restarts repeatedly. Check available disk space with df -h. ClickHouse requires sufficient space for the kasm_clickhouse Docker volume and exits when the disk is full.
  • Collector container exits immediately after starting. Run docker logs kasm_otel_collector before the container shuts down. A configuration syntax error in /opt/kasm/current/conf/otel/otel-collector-config.yaml prevents startup. Restore the file from a backup or re-run the installer to regenerate it.
  • Telemetry data appears for some nodes but not others in a distributed deployment. This usually indicates a token mismatch on the silent nodes. Follow the steps in Resolve remote-collector token mismatches.