Recover from quorum loss

Overview

An OpenZiti high-availability cluster relies on Raft quorum to make decisions. A cluster loses quorum when more than half of its configured members become unreachable. The surviving controllers then become read-only, and the cluster cannot accept changes. This guide restores quorum with kziti deploy ha recover after a controller goes down without clean removal, so the cluster returns to normal write operation while edge state is preserved.

Quorum loss most often follows a controller that fails before it can be cleanly removed from membership. Common causes include hardware failure, a terminated VM, or a network partition that does not heal. The recovery command spins up a short-lived ephemeral peer that impersonates the dead member to restore quorum, runs the remove-member step, then cleans up. Edge state such as users, services, routers, and policies is preserved.

caution

This procedure recovers from a node that is permanently gone or not coming back. If the missing node is only temporarily unreachable, such as a network blip, host restart, or planned maintenance, wait for it to return rather than running recovery. Removing a node that later rejoins creates split-brain risk.

Prerequisites

Before you begin, confirm the following:

• A working kziti deployment.
• At least one controller in the cluster is still reachable. You run the recovery command on that host.
• The node name of the dead member, which is the value of --node-name from when it was originally added.

Solution approach

This guide progresses through the following phases:

1. Confirm quorum is lost.
2. Run recovery.
3. Verify write quorum is restored.
4. Restore the original HA topology.

Detailed steps

Confirm quorum is lost

1. On a surviving controller host, list the cluster members:

   docker compose -f /opt/kziti/docker-compose.yml exec ziti-controller \
     ziti agent cluster list

   Quorum is lost if the dead member appears as voter: true and the cluster is unable to elect a leader. Read access typically still works, so listing identities and services succeeds. Mutating commands fail with timeouts or quorum errors.

Run recovery

1. On a surviving controller host, run the recovery command:

   kziti deploy ha recover --node ziti-c-2

   --node is the dead member's node name. The command performs the following steps:

   1. Reads cluster state to confirm ziti-c-2 is still a member.
   2. Starts an ephemeral peer container with the same node identity, joining as a non-voter.
   3. The cluster regains quorum once the ephemeral peer is connected.
   4. remove-member ziti-c-2 runs against the cluster leader.
   5. The ephemeral peer is torn down.

   The command typically completes in under a minute.

Verify write quorum is restored

1. List the cluster members again:

   docker compose -f /opt/kziti/docker-compose.yml exec ziti-controller \
     ziti agent cluster list

   The dead node should be gone from the member list. The cluster should report a healthy leader, and the remaining controllers should all be voters.

2. Run a mutating operation to confirm write quorum is restored. Read-only commands still work even when quorum is lost, so they do not prove recovery:

   kziti network create quorum-test "Quorum Test" && \
     kziti network delete quorum-test --yes

   If both commands complete without error, the cluster is accepting writes and is back to normal operation.

Restore the original HA topology

1. Recovery restores quorum, but the cluster now runs with one fewer controller than the target topology. Add a new HA controller on a fresh host, or on the recovered host once you have repaired it.

Common troubleshooting steps

• The dead node still appears as a voter after recovery. Run ziti agent cluster list again, because membership changes propagate through the cluster leader. Confirm that kziti deploy ha recover returned without error on the surviving controller host.
• Mutating commands still fail after recovery. Confirm that the cluster reports a healthy leader and that the remaining controllers are voters. Read-only commands succeed even without quorum, so verify with kziti network create and kziti network delete.
• You are unsure whether the node is truly gone. Run recovery only for a node that is permanently gone. A node that later rejoins after removal creates split-brain risk. Wait for a temporarily unreachable node to return instead.