Skip to main content
Version: Developer

Add an HA controller

Overview​

A single OpenZiti controller is a single point of failure for a kziti deployment. This guide expands a healthy single-controller deployment into a high-availability cluster by joining additional controllers that share state through Raft consensus. You generate an encrypted join package on the existing controller, transfer it to a new host, and run the bundled installer. The new controller then synchronizes Raft state and promotes itself to a voting member, so the cluster survives the loss of a single host.

Prerequisites​

Before you begin, confirm the following:

  • A healthy single-controller kziti deployment. Production clusters commonly use three nodes, where a 2-of-3 quorum survives the loss of one host.
  • Each new controller host has docker and docker compose installed, matching the bootstrap host.
  • Each new controller host has 7zip installed to extract the encrypted join package. Install p7zip-full on Debian or Ubuntu, or p7zip on RHEL or Fedora.
  • A DNS A record for each new controller, for example ziti2.example.com, and a separate record for its router, for example ziti2-router.example.com.
  • Inbound ports 1280/tcp, 6262/tcp, and 3022/tcp open on each new controller host.
  • A separate secure channel, such as a password manager or an encrypted message, to deliver the extraction password apart from the join package.
note

The new host does not need the kziti binary. The join package is self-contained and includes the installer.

Solution approach​

This guide progresses through the following phases:

  1. Generate and transfer the join package from the existing controller.
  2. Install the new controller from the join package.
  3. Verify cluster health.

Repeat phases one and two for each additional controller you want to add.

Detailed steps​

Generate and transfer the join package​

Perform these steps on the existing controller host.

  1. Generate a join package for the new controller. Replace the placeholder values with the new node name, the new controller and router FQDNs, and your administrator password.

    kziti deploy ha create-join-package \
      --output /tmp/join.zip \
      --node-name ziti-c-2 \
      --controller-host ziti2.example.com \
      --router-host ziti2-router.example.com \
      --admin-password '<your-admin-password>'

    The flags control how the new node joins the cluster:

    • --node-name is the cluster member name. It must be unique within the cluster.
    • --controller-host and --router-host are the FQDNs of the new controller and its router, not the existing one.
    • Add --controller-ip <new-controller-ip> if the new controller's hostname does not resolve from inside the existing controller container. This injects an extra_hosts entry into a docker-compose override on the existing host, so the existing host reaches the new node by name during the join.

  2. Note the output. kziti produces an AES-encrypted zip at the path you supplied with --output, and prints a randomly generated extraction password.

  3. Transfer the zip to the new host.

    scp /tmp/join.zip user@ziti2.example.com:/tmp/

  4. Send the extraction password through a separate secure channel, such as a password manager or an encrypted message. Do not put the password in the same email or chat as the zip.

Install the new controller​

Perform these steps on the new controller host.

  1. Install 7zip if it is not already present.

    # Debian / Ubuntu
    apt-get install -y p7zip-full

    # RHEL / Fedora
    dnf install -y p7zip

  2. Extract the encrypted package and run the bundled installer. Replace <extraction-password> with the password from the previous phase.

    7z x -p<extraction-password> /tmp/join.zip -o/tmp/kziti-join
    bash /tmp/kziti-join/install.sh

    If the bootstrap controller's hostname does not resolve from this new host, add --existing-controller-ip <bootstrap-ip> to the install.sh invocation. This lets the new node reach the bootstrap controller during the join.

  3. Allow the new controller to join. The new controller starts as a non-voter, syncs Raft state from the existing cluster, and self-promotes to a voting member once it is caught up.

Verify cluster health​

  1. On any controller host, list the cluster members.

    docker compose -f /opt/kziti/docker-compose.yml exec ziti-controller \
      ziti agent cluster list

  2. Confirm that the new node appears with voter: true. Promotion typically completes within a minute of the join.

Common troubleshooting steps​

  • The new node never reaches voter: true. Confirm that ports 1280/tcp, 6262/tcp, and 3022/tcp are open inbound on the new host and that Raft state is syncing. The node remains a non-voter until it catches up with the existing cluster.
  • The join fails because hostnames do not resolve. Add --controller-ip <new-controller-ip> to kziti deploy ha create-join-package on the existing host, or add --existing-controller-ip <bootstrap-ip> to the install.sh invocation on the new host.
  • Extraction of the join package fails. Confirm that 7zip is installed on the new host and that the extraction password matches the one printed by kziti. The password and the zip travel through separate channels, so verify that you used the correct password.
  • The new node uses a duplicate name. Set a unique --node-name for each controller. A name that already exists in the cluster prevents the new node from joining.