Healthcheck system
The Kasm Workspaces healthcheck system is a container-based monitor that validates the health of each Kasm Workspaces Docker service.
Each service container runs /usr/bin/healthcheck.sh, which queries a service-specific healthcheck endpoint and, optionally, verifies that a TCP port is listening. The endpoint returns a JSON response that reports service status. The script reports the result through its exit code, which Docker reads to set container health state.
The script also decides when to restart a service that stays unhealthy, because Docker Compose does not act on the unhealthy state. It signals the service process directly.
The API Server additionally runs a dedicated liveness endpoint that takes over its restart decision. See API Server liveness endpoint. Every other service uses its healthcheck endpoint for both the reported health state and the restart decision.
Healthcheck endpoint response
Each service exposes a standardized healthcheck endpoint that returns a JSON response indicating service status. The following command queries the endpoint through the public Kasm Workspaces address:
curl -k https://<kasm-server>/api/__healthcheck
The endpoint returns the following response when the service is healthy:
{"ok": true}
Service endpoints
The following table lists the healthcheck endpoint and TCP port used for each Kasm Workspaces service as shown in the example configurations. Confirm exact paths and ports for a specific deployment against the shipped Compose configuration.
| Service | Endpoint path | Port check |
|---|---|---|
| Kasm API | /api/__healthcheck | none |
| Kasm Manager | /__healthcheck | none |
| Kasm Agent | /__healthcheck | none |
| Kasm Connection Proxy (guac) | /__healthcheck | none |
| Kasm Connection Proxy (RDP gateway) | /__healthcheck | 3389 |
| Kasm Connection Proxy (RDP over HTTPS gateway) | /rdp-https-gateway/__healthcheck | none |
Healthcheck script parameters
The container healthcheck system runs /usr/bin/healthcheck.sh with the following parameters:
/usr/bin/healthcheck.sh <target_url> <port> <retry_delay> <max_delay> <healthcheck_timeout> <kill_threshold> <liveness_url> <restart_grace> <liveness_threshold>
The script accepts positional arguments. Trailing arguments are optional and fall back to their default values when omitted in order.
| Parameter | Description | Example | Required | Default |
|---|---|---|---|---|
target_url | HTTP or HTTPS endpoint to validate service health. | http://localhost:8080/api/__healthcheck | Yes | none |
port | TCP port to verify is listening. Use "" to skip the port check. | 3389 or "" | No | none |
retry_delay | Initial delay before the next check after a failure, in seconds. The delay doubles on each consecutive failure, up to max_delay. | 5 | No | 1 |
max_delay | Maximum delay for exponential backoff, in seconds. | 60 | No | 30 |
healthcheck_timeout | Maximum time allowed for the TCP port check and the liveness request, in seconds. The target_url request uses a fixed five-second timeout. | 5 | No | 5 |
kill_threshold | Consecutive target_url failures required before the script signals a restart. | 2 | No | 2 |
liveness_url | Dedicated liveness endpoint. Set only for the API Server. See API Server liveness endpoint. | http://localhost:8081/livez | No | none, disabled |
restart_grace | Time the service has to shut down gracefully after SIGUSR2 before the script sends SIGKILL, in seconds. | 30 | No | 30 |
liveness_threshold | Consecutive liveness_url failures required before the script signals a restart. Applies only when liveness_url is set. | 3 | No | 3 |
Restart behavior
A single failure never restarts a service. The script counts consecutive failures and acts only after the applicable threshold is reached: kill_threshold for the target_url check, or liveness_threshold for the liveness check. Any success resets the count.
Once a threshold is reached, the script asks the service to shut down gracefully by sending SIGUSR2 to the service process. If the process is still running restart_grace seconds later, the script sends SIGKILL. Docker then restarts the container under its configured restart policy.
The script persists its retry schedule, failure counts, and pending restart state under /healthcheck, a bind mount that outlives the container. Each service container mounts its own directory, so state does not carry between services.
Example configurations
The Kasm API service skips the port check, uses a five-second retry delay with a 60-second maximum delay, and sets a liveness endpoint:
/usr/bin/healthcheck.sh http://localhost:8080/api/__healthcheck "" 5 60 5 2 http://localhost:8081/livez
The Kasm Manager service skips the port check:
/usr/bin/healthcheck.sh http://localhost:8181/__healthcheck "" 5 60
The Kasm Connection Proxy (RDP gateway) verifies that port 3389 is listening:
/usr/bin/healthcheck.sh http://localhost:5555/__healthcheck 3389 1 30
The Kasm Agent service skips the port check, uses longer retry and maximum delays, and allows 15 seconds for the check itself:
/usr/bin/healthcheck.sh http://localhost:4444/__healthcheck "" 30 90 15
API Server liveness endpoint
The API Server is the only service configured with a liveness_url. This section applies to that service alone.
/api/__healthcheck validates that the API Server can reach the database, and it travels through the same request worker pool as production traffic. That makes it a good signal for a "readiness probe".
The API Server also serves a second endpoint, GET /livez on port 8081, that reports only whether the request worker pool is still making forward progress. A separate thread answers it, independent of the worker pool and the database connection pool, so neither load nor a slow database can delay a response. A service under heavy load keeps completing requests, just slowly, and reports healthy; a deadlocked service completes none, and reports unhealthy.
With liveness_url set, the two checks divide responsibility:
| Concern | Endpoint used |
|---|---|
| Health state reported to Docker | /api/__healthcheck |
| Restart decision | /livez |
A failing /api/__healthcheck therefore marks the container unhealthy in docker ps but never restarts it. kill_threshold has no effect on this service.
Port 8081 is internal to the container and the Docker network. It is not published to the host and not proxied through the public address.
Liveness endpoint response
The endpoint returns text/plain. A healthy service returns HTTP 200 with the following body:
healthy
A deadlocked service returns HTTP 503 and a body that states the reason, for example:
unhealthy: no request completed in 94s while in_flight=20 queued=13
The script treats each outcome as follows:
| Outcome | Meaning | Effect |
|---|---|---|
HTTP 2xx | The worker pool is making progress. | Clears the liveness failure count. |
HTTP 503 | The endpoint reports a deadlock. | Counts toward liveness_threshold. |
| No response | The process is frozen, or the request exceeded healthcheck_timeout. | Counts toward liveness_threshold. |
| Connection refused | Nothing is listening, so the service is still starting. | Never restarts the service. |
HTTP 4xx | The probe URL is misconfigured. A wrong path returns 404, a wrong method 405. | Never restarts the service. |
Liveness tuning
Two settings in api.app.config.yaml (/opt/kasm/current/conf/app/api/api.app.config.yaml) tune the liveness endpoint. Change them only to resolve a port conflict or to accommodate a deployment with unusually long request latency.
| Setting | Description | Default |
|---|---|---|
server.liveness_port | Port the liveness endpoint listens on. Update liveness_url in the healthcheck configuration to match. | 8081 |
server.liveness_stall_seconds | Time the worker pool may complete zero requests, while requests are in flight or queued, before reporting 503. | 90 |
Each setting can also be overridden per-container with an environment variable (KASM_LIVENESS_PORT, KASM_LIVENESS_STALL_SECONDS) without editing the config file.
Set server.liveness_stall_seconds well above the highest legitimate request latency in the deployment. A value below that latency reports a busy service as deadlocked.
Kubernetes probes
In a Kubernetes deployment, point an httpGet liveness probe at /livez on port 8081 and set failureThreshold to the same value used for liveness_threshold elsewhere in the deployment. Point a readiness probe at /api/__healthcheck so that a dependency failure removes the pod from service without restarting it.
Because /livez does not bind until the application finishes starting, use a startupProbe to bound startup rather than a long initialDelaySeconds on the liveness probe.
Docker healthcheck considerations
Standard Docker healthcheck parameters (interval, timeout, retries, and start_period) apply to all services. Set the Docker timeout parameter to a value no less than the script's internal healthcheck_timeout. A lower Docker timeout ends the check before the script completes.
The Docker retries parameter controls only the reported health state. It does not affect restarts, which kill_threshold and liveness_threshold govern.
Related references
- System Metrics describes the dashboard that reports overall deployment health across Kasm Workspaces components.