`ctx` Hub: High-Availability Cluster¶

Run multiple hub nodes with Raft-based leader election for redundancy. Any follower can take over if the leader dies.

This recipe assumes you've read the ctx Hub overview and the Multi-machine setup. HA only makes sense in the "small trusted team" story; a personal cross-project brain on one workstation does not need three Raft peers.

Raft-Lite

ctx uses Raft only for leader election, not for data consensus. Entry replication happens via sequence-based gRPC sync on the append-only JSONL store. This is simpler than full Raft log replication and is possible because the store is append-only and clients are idempotent. The implication: a write accepted by the leader is durable on the leader immediately; followers catch up asynchronously. If the leader crashes between accepting a write and replicating it, that write can be lost. Do not use the hub as a bank ledger.

Topology¶

A minimum HA cluster is three nodes. Two is worse than one: it doubles failure probability without providing quorum.

         +-------------+
         |  client(s)  |
         +------+------+
                |
    +-----------+-----------+
    |           |           |
+---v---+   +---v---+   +---v---+
| hub A |   | hub B |   | hub C |
| :9900 |   | :9900 |   | :9900 |  gRPC (clients, data sync)
| :9901 |   | :9901 |   | :9901 |  Raft (leader election)
+-------+   +-------+   +-------+
    ^           ^           ^
    +-----------+-----------+
        Raft (leader election)
        gRPC (data sync)

Each node runs two listeners: the hub's gRPC port that clients dial (--port), and the Raft port the other nodes dial (--raft-bind). They are separate addresses; the peer list is made of Raft addresses.

Step 1: Bootstrap the First Node¶

ctx hub start --daemon \
  --port 9900 \
  --raft-bind hub-a.lan:9901 \
  --peers hub-b.lan:9901,hub-c.lan:9901

--raft-bind is the address this node advertises to the other two, so it has to be a host they can dial: a bare port (:9901) or a wildcard (0.0.0.0:9901) is rejected at startup. Every node's --raft-bind appears in the other nodes' --peers lists, and each node bootstraps that same set.

The node starts a Raft election as soon as it sees its peers. Until a quorum answers, ctx hub status reports Leader: unknown (election in progress) — expected while the other nodes are still coming up.

Step 2: Start the Other Nodes¶

On hub-b.lan:

ctx hub start --daemon \
  --port 9900 \
  --raft-bind hub-b.lan:9901 \
  --peers hub-a.lan:9901,hub-c.lan:9901

On hub-c.lan:

ctx hub start --daemon \
  --port 9900 \
  --raft-bind hub-c.lan:9901 \
  --peers hub-a.lan:9901,hub-b.lan:9901

After a few seconds, one node wins the election and becomes the leader. The other two are followers.

Step 3: Verify Cluster State¶

From any node:

ctx hub status

Expected output on the node that won the election:

Role: Leader
Leader: hub-a.lan:9901
Entries: 1248  Peers: 2

and on either of the others:

Role: Follower
Leader: hub-a.lan:9901
Entries: 1248  Peers: 2

The leader is named by its Raft address, which is what the cluster agrees on. Clients still dial the hub port.

Peers: counts the servers in the committed Raft configuration other than the one answering, so a three-node cluster reports two from every node. If the line reads Leader: unknown (election in progress), Raft has no leader for the current term: either the election is still running, or the node you asked cannot see a quorum.

Step 4: Register Clients with Failover Peers¶

The ctx hub * commands above run on the hub nodes themselves and don't need a project. The ctx connection * commands below are different: they live inside a project (the encrypted hub config is stored at .context/.connect.enc), so you have to tell ctx which project first.

When registering a client, give it the full peer list:

# In the project directory on the client:
ctx connection register hub-a.lan:9900 \
  --token ctx_adm_... \
  --peers hub-b.lan:9900,hub-c.lan:9900

If the leader becomes unreachable, the client reconnects to the next peer. Followers redirect to the current leader, so writes always land on the right node.

Runtime Membership Changes¶

Membership changes are admin-gated and leader-only: pass --token (or set CTX_HUB_ADMIN_TOKEN) and run them against the leader that ctx hub status names. A follower answers not the leader rather than pretending.

Adding a node is two steps, because a new node must not bootstrap a configuration of its own — it waits for one:

# On hub-d.lan, the new node:
ctx hub start --daemon \
  --port 9900 \
  --raft-bind hub-d.lan:9901 \
  --join

# On the leader:
ctx hub peer add hub-d.lan:9901 --token ctx_adm_...

ctx hub status on hub-d.lan then reports Role: Follower and names the leader; every node's Peers: count goes up by one.

Remove a decommissioned peer, again on the leader:

ctx hub peer remove hub-c.lan:9901 --token ctx_adm_...

Removal shrinks the quorum, which is the point: a node you have taken away should stop counting against liveness. Removing one of three leaves two, and a two-node cluster needs both to be up — plan the next addition accordingly.

Planned Maintenance¶

Before taking a leader offline, hand off leadership:

ssh hub-a.lan 'ctx hub stepdown --token ctx_adm_...'

stepdown asks Raft to transfer leadership to a follower that is caught up, and returns once the transfer completes. Run ctx hub status afterwards to see which node won. Then stop the old leader; the cluster keeps serving throughout, because the handoff happened before the process went away.

Failure Modes at a Glance¶

Event	What happens
Leader crashes	New election; clients reconnect to new leader
Follower crashes	No write impact; catches up on restart
Network partition (majority)	Majority side keeps serving; minority read-only
Network partition (split)	No quorum; all nodes read-only
Disk full on leader	Writes rejected; read traffic continues

For the full list, see Hub failure modes.

ctx Hub: High-Availability Cluster¶