SIEM and log forwarding

Overview

Kasm Workspaces 1.20.0 routes all log and telemetry forwarding through the OTel Collector, replacing the legacy Splunk and Elasticsearch forwarding settings that were available in earlier versions. Administrators can export telemetry to Splunk via the Splunk HEC exporter, to any OTLP-compatible backend via HTTP forwarding, or to a file-based log shipper by reading the structured JSON logs that each Kasm service writes to disk. This guide explains all three forwarding paths and shows how to query ClickHouse directly when the built-in database is in use.

Prerequisites

• A running Kasm Workspaces 1.20.0 or later deployment with the observability stack enabled.
• Root access to the host running kasm_otel_collector.
• For Splunk HEC forwarding: a Splunk deployment with an HEC input and a valid HEC token.
• For OTLP/HTTP forwarding: an endpoint URL and authentication token from your backend provider.
• For file-based forwarding: Filebeat, Splunk Universal Forwarder, or equivalent installed and configured to read from /opt/kasm/current/log/.

Solution approach

This guide progresses through the following phases:

1. Configure Splunk HEC export from the collector.
2. Configure OTLP/HTTP forwarding to Grafana Cloud, Splunk OTLP ingest, or another OTLP backend.
3. Configure file-based log forwarding using Filebeat or the Splunk Universal Forwarder.
4. Query ClickHouse directly for log analysis and migration from the legacy logs table.

Detailed steps

Configure Splunk HEC export

The collector configuration at /opt/kasm/current/conf/otel/otel-collector-config.yaml contains commented template blocks for Splunk HEC export.

1. Open the configuration file:

   sudo nano /opt/kasm/current/conf/otel/otel-collector-config.yaml

2. Locate the block between # SPLUNK_HEC_EXPORTER_START and # SPLUNK_HEC_EXPORTER_END. Uncomment that block and fill in the following values:

   Value	Description
   endpoint	Splunk HEC endpoint URL (for example, https://splunk-host:8088/services/collector)
   token	Splunk HEC token
   source	Pre-set to kasm; change only if your Splunk configuration requires it
   sourcetype	Pre-set to kasm:otel; change only if your Splunk configuration requires it

3. Locate the block between # SPLUNK_HEC_PIPELINE_START and # SPLUNK_HEC_PIPELINE_END. Uncomment that block.

4. Restart the collector to apply the changes:

   docker restart kasm_otel_collector

note

When upgrading from a Kasm Workspaces version that had Splunk forwarding configured in Global Settings, the upgrade script migrates those settings into the Splunk HEC exporter block automatically. Any legacy Splunk token prefix is stripped during migration. After upgrading, verify the exported configuration and enable the block as described in steps 2 through 4 above.

Configure OTLP/HTTP forwarding

Use the --otel-forward-endpoint and --otel-forward-token flags at install or upgrade time to forward all telemetry signals to an additional OTLP/HTTP endpoint. Forwarding is additive: the local ClickHouse pipeline continues to receive all telemetry regardless of the forwarding configuration.

Grafana Cloud example:

1. In your Grafana Cloud portal, navigate to OpenTelemetry then Configure and generate an access token. The token is a Base64-encoded instanceId:apiKey string.

2. Run the installer with the endpoint and token:

   sudo bash install.sh --role all \
     --otel-forward-endpoint "https://otlp-gateway-prod-us-east-3.grafana.net/otlp" \
     --otel-forward-token "YOUR_GRAFANA_CLOUD_TOKEN"

   The token is sent as a Bearer token in the Authorization header.

Splunk OTLP ingest example:

sudo bash install.sh --role all \
  --otel-forward-endpoint "https://ingest.YOURSPLUNKDOMAIN.signalfx.com/v2/datapoint/otlp" \
  --otel-forward-token "YOUR_SPLUNK_ACCESS_TOKEN"

Configure file-based log forwarding

All Kasm application services write structured JSON log files to /opt/kasm/current/log/. Point Filebeat, the Splunk Universal Forwarder, or any file-based shipper at these paths.

Application service log files:

File	Service	Format
/opt/kasm/current/log/api.log	API Server	JSON
/opt/kasm/current/log/agent.log	Provision Agent	JSON

nginx proxy log files:

File	Content	Format
/opt/kasm/current/log/nginx/access_json.log	HTTP access log (all requests)	JSON
/opt/kasm/current/log/nginx/error.log	nginx error log	Text

The nginx access log (access_json.log) is a dual-write file: nginx writes it to disk, and the OTel Collector's filelog receiver also reads it. Both destinations receive identical data.

Log file names consolidated in 1.20.0

Each application service now writes a single JSON log file. Update shipper paths that reference the earlier files.

File in 1.19.x	File in 1.20.0	Notes
api_server.log	api.log	Renamed. JSON instead of plain text
api_server_json.log	api.log	Merged
admin_api_server.log, client_api_server.log, subscription_api_server.log, manager_api_server.log, manager_api_server_json.log	api.log	Merged. The manager runs inside the API server process
agent.log	agent.log	Same name, now JSON instead of plain text
agent_json.log	agent.log	Merged

The logging section of api.app.config.yaml and agent.app.config.yaml is no longer read. Rotation and file location are controlled by the KASM_LOG_FILE_* environment variables. See Configure log file rotation. A syslog handler address set in the previous api.app.config.yaml is migrated into the OTel Collector syslog exporter by the upgrade script. That exporter forwards records from every service, including the agent. The upgrade does not read agent.app.config.yaml, so an address set only there, or one that differs from the API server's, is not carried over. In that case, uncomment the syslog exporter and the logs/syslog pipeline in /opt/kasm/current/conf/otel/otel-collector-config.yaml on the all or app node, set the host and port, and restart the services.

JSON field names updated in 1.20.0

The application log JSON field names have changed in 1.20.0. Update any existing parsers to use the new names.

Old field	New field	Notes
levelname	level	Renamed
name	logger	Renamed
lineno	line	Renamed
asctime	(removed)	Use timestamp instead
processName	(removed)	No replacement
threadName	(removed)	No replacement
filename	(removed)	No replacement
funcName	(removed)	No replacement
timestamp	timestamp	Unchanged
module	module	Unchanged
message	message	Unchanged

The JSON log files carry only these six fields: timestamp, level, logger, module, line, and message. To add OTel log attributes such as kasm.metric_name or kasm.kasm.user_name to the files, set KASM_LOG_FILE_EXTRA_FIELDS on the service container. The file writes each name with underscores, so kasm.kasm.user_name appears as kasm_kasm_user_name. The full attribute set is always present in ClickHouse and in the collector forwarders. For the variable, see Observability configuration. For the attribute names, see Session lifecycle events.

nginx access log field mapping in ClickHouse

The OTel Collector renames nginx access log fields before storing them in ClickHouse. Use the following names when querying ClickHouse directly.

nginx field	OTel attribute in ClickHouse
request_method	http.request.method
uri	url.path
http_status	http.response.status_code

Query ClickHouse directly

All telemetry is stored in the kasm_otel ClickHouse database. Connect using the ClickHouse client:

docker exec -it kasm_clickhouse clickhouse-client -u kasm_otel --password '<password>'

Key tables:

Table	Contents
kasm_otel.otel_logs	All log records
kasm_otel.otel_traces	All spans
kasm_otel.otel_metrics_sum	Counter metrics
kasm_otel.otel_metrics_gauge	Gauge metrics

Example: query recent login failures:

SELECT Timestamp, LogAttributes['kasm.kasm.user_name'] AS username, Body
FROM kasm_otel.otel_logs
WHERE LogAttributes['kasm.metric_name'] = 'kasm.account.login.failed'
  AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC;

To query nginx proxy records, filter by ResourceAttributes['service.name'] = 'kasm-proxy'.

Mapping from the legacy PostgreSQL log table

Use this table when migrating queries from the legacy PostgreSQL logs table to ClickHouse.

Old logs column	otel_logs expression
ingest_date	Timestamp
message	Body
levelname	SeverityText
host	ResourceAttributes['host.name']
metric_name	LogAttributes['kasm.metric_name']
kasm_user_name	LogAttributes['kasm.kasm.user_name']
kasm_id	LogAttributes['kasm.kasm.id']
server_id	LogAttributes['kasm.server.id']
client_ip	LogAttributes['client.address']

Common troubleshooting steps

• Splunk HEC not receiving data. Confirm the HEC pipeline block is uncommented and the collector was restarted. Check docker logs kasm_otel_collector for export errors mentioning the HEC exporter.
• OTLP forwarding fails with authorization errors. Verify the token format matches your provider's requirements. For Grafana Cloud, confirm the token is a valid Base64-encoded instanceId:apiKey string.
• File-based shipper cannot read log files. Confirm the shipper process has read access to /opt/kasm/current/log/ and that the log files exist.
• ClickHouse queries return no results after upgrading to 1.20.0. The logs table in PostgreSQL is no longer used. Query kasm_otel.otel_logs in ClickHouse instead, using the column mapping table above.