Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Replication

hoardDB replicates data between nodes over TCP+TLS. This document is the design record: the model, the wire path, what is guaranteed, and what is not implemented yet.

Status: working and tested for the model below — three nodes at RF=3 and five nodes at RF=3 (exactly three of the five hold each bucket), data verified present in each node’s own storage, write concern enforced, and a restarted replica repaired from the owner’s log. See server/cluster_replication_test.go.

What was here before

The previous implementation reported success without replicating anything. replicateEntryAsync dialled a peer, logged that it was sending append entries, and then set peer.lastAckSeq = entry.Seq on its own and signalled a local acknowledgement. WaitForAcks returned len(peerIDs) after a single local signal, so even w=all was satisfied by the leader alone. Nothing called it, either: no handler went through the write path, and storage.OpenWAL was never called, so the WAL was never open.

Two consequences worth remembering:

• A durability guarantee that is not backed by a network round trip is worse than no guarantee, because a client will trust it.
• Opcode.IsMutating() answers “is this unsafe to process from 0-RTT?”, not “does this change stored state?”. It returns false for OpPut and OpBatchPut because they are idempotent overwrites. Routing by that predicate silently exempts every data write from replication. Route by !IsReadOnly() instead.

Model

Per-bucket owner with ring-placed replicas.

• A bucket’s owner is chosen by hashing the bucket name onto the ring (ring.Ring.Owner). The owner is the serialisation point for writes to that bucket, and the node that names the replication sequence.
• The replica set is the owner plus the next replication_factor - 1 distinct nodes clockwise (ring.Ring.Replicas). With RF=3 and three members, every node holds every bucket.
• Reads are served by any node in the replica set.

Membership is static: every node is configured with the same member set via HOARDB_SEED_NODES, and each derives the ring locally. There is no automatic membership change. Because placement must agree across nodes, the member sets are compared on every gossip exchange, and a disagreement makes the node refuse mutations rather than scatter writes across replica sets its peers disagree with.

A node’s ring identity is the address it is listed under; HOARDB_NODE_ID is only needed when you want an identity that is not the address (written as id@host:port). Node identity is validated at startup: a node whose identity is not on the ring is a configuration error and startup fails with the member list rather than running in a state where it would never take ownership of anything.

Write path

client ──▸ coordinator (any node)
              │  not the owner? forward the entry
              ▼
            owner
              1. append to WAL (fsync)      ← the entry is durable here
              2. apply locally
              3. ship the entry to the other RF-1 replicas
              4. wait for the write concern
              5. answer the client
replica ──▸ 1. append to WAL (fsync)
            2. apply locally
            3. acknowledge the sequence number

An entry is the original client opcode plus its original request payload (ReplEntry). A replica applies an entry through the same Server.dispatch call a local client request takes, so a mutation has exactly one implementation and cannot drift between the client path and the replication path.

Write concern is evaluated against confirmed sequence numbers, never against local state:

• one — the owner’s own apply. No replica wait.
• majority — RF/2 + 1 copies, the owner counts as one.
• all — RF copies.

If the required number of replicas do not confirm within the timeout (5s, DefaultWriteConcernTimeout), the client gets an error naming how many copies did confirm(server.ErrWriteConcernTimeout). A write that could not be replicated is never reported as replicated. If the replica set is smaller than the write concern requires — for example RF=3 with only one node running — the write fails with server.ErrReplicaSetIncomplete rather than quietly storing one copy.

Write concern can be set per request ("write_concern": "majority" on an insert or batch insert, HOARDB_WRITE_CONCERN for the node default). The default with RF>1 is majority.

Wire protocol

No new opcodes were needed. The cluster opcodes already in the protocol spec are used:

• GOSSIP (0x90) — announce this node, exchange member lists, verify agreement. Also the join handshake.
• REPL_PUSH (0x91) — apply these entries. One frame serves both hops: with return_response the owner’s response is relayed to the client (coordinator → owner), without it the acknowledgement is the applied-through sequence (owner → replica).
• REPL_ACK (0x92) — the response opcode for a replication push.

Node-to-node sessions authenticate with the cluster token, which already exists on every node and is exactly the credential that authorises cluster membership (HOARDB_CLUSTER_TOKEN, or the cluster.token file in the data directory). The transport rejects every non-auth opcode on an unauthenticated session, so a node authenticates before it can replicate. Tokens are compared in constant time against an in-memory copy; the file is re-read only when the presented token does not match, which covers rotation.

A failed write returns a defined error response rather than a handler error: the transport turns a handler error into a fatal stream reset, which would tear down a client’s session over something as ordinary as an unmet write concern.

Durability, recovery and repair

• The WAL is the authority for what a node has durably accepted. Every entry is appended before it is applied, on the owner and on every replica.
• On startup a node replays its WAL through the ordinary dispatch path. Replay is best-effort per entry (a create for a bucket that already exists is expected, not fatal) and runs once per process.
• A replica that missed entries while it was down is repaired from the owner’s log when it comes back: the owner replays the entries the peer has not confirmed, filtered to the buckets that peer actually holds. The acknowledgement position advances only as far as the peer says it applied.
• Replication is at-least-once: an acknowledgement lost in flight makes the sender re-send, so a receiver never applies the same sequence twice (Cluster.applyDedup). Without that, non-idempotent store types (FIFO, LIFO, heap, btree) would silently gain duplicate elements.
• A failed send is retried, never dropped. This is not a nicety: the acknowledgement is a high-water mark, so an entry that is silently discarded while a later one is acknowledged becomes permanently invisible — catch-up starts above it, and the deduplication guard rejects it if it is ever sent again. Measured before the fix: with one replica of three down, 2 of 10 documents written at majority never arrived on that replica, which the ring still counted as one of the three copies. The sender now retries the same batch until the peer accepts it and does not drain the queue while a batch is outstanding, and catch-up refuses to start above an entry still in flight.

Limitations

These are real and worth knowing before relying on the cluster:

• Eventual consistency by default. A read is served from whichever replica the client reaches. A replica that is behind (or was just offline and has not been repaired yet) can return stale or missing data. A midpoint reads as locally-consistent-per-replica, not globally consistent.

  Opt-in strong consistency: consistent get returns the freshest copy by reading a majority of holders and repairs a stale replica through the normal replication path. The ordinary get is unchanged — still eventual, still served from whichever replica is reached.

• Database-level DDL is not covered by write concern. create database has no bucket to place by, so it is broadcast to every member on the same ordered queue as bucket writes and not waited for. Ordering guarantees a bucket created after its database arrives after it; acknowledgement does not.

• The WAL is truncated only up to what every replica has confirmed. WAL.Truncate can be called, but the server’s retention pass (server/retention.go, every HOARDB_WAL_RETENTION_INTERVAL seconds) prunes only segments every replica of this node’s buckets has confirmed and every local store has been synced past, and it syncs the Badger stores first. A replica that is down simply keeps the log from shrinking while it is down — which is exactly when the log is the repair tool. WAL.FirstSequence() is the retained horizon, and catch-up refuses (rather than silently skipping) any peer whose position is below it. (Before retention existed the WAL was never truncated and this footgun was latent.)

• A brand-new node cannot bootstrap from nothing. Catch-up replays what the owner still holds in its log; there is no snapshot transfer, so an empty node joining an existing cluster stays empty. Add it with an empty data directory only into an empty cluster.

• Exactly-once is not guaranteed across a restart. The deduplication guard is in memory; after a restart a re-sent entry can be applied twice. Hash, heap and blob-key stores are idempotent by key, so this only affects append-style store types (FIFO, LIFO, btree).

• Static membership. Adding or removing a node means changing configuration on every node and restarting. Placement changes with membership, so data is not moved automatically for you.

• The CLI does not set write concern per request yet, even though requests carry the field.

• A node outside a bucket’s replica set does not hold the bucket at all. Bucket DDL is placed on the replica set rather than broadcast, so show buckets on a non-replica node lists nothing for it even though reads and writes work there (they are forwarded). With 5 members and RF=3 that is two of five nodes.

• Route reads to the holders, not to every node. A forwarded read costs about 2.4× a local one (measured: 5,024/s forwarded against 11,927/s local at one read in flight), so a client that round-robins over all five members of a 5-node RF=3 cluster is slower than one that uses a single node: 65,820/s versus 74,105/s, while round-robining over the three holders gives 95,602/s (+29%). Anything in front of the port — a load balancer, a proxy, DNS — must know which nodes hold a bucket, which is why clients are expected to use the ring rather than a generic balancer. Details: bench/RESULTS.md.

Testing

server/cluster_replication_test.go starts three real TCP+TLS servers on loopback with a real ring and a real cluster token — nothing is mocked, because only an end-to-end test can show that data actually arrives.

• TestClusterPropagatesWritesToEveryReplica writes 30 documents to one node and reads each one back from every node’s own storage.
• TestWriteConcernFailsWhenReplicasAreDown stops two of three nodes and asserts majority and all fail while one still succeeds.
• TestRestartedReplicaCatchesUp stops a replica, writes at majority, restarts it, and asserts the missed writes are repaired from the owner.
• TestReplicaReceivesEntriesRetriedAfterAFailedSend stops a replica, writes at majority, restarts it, and asserts the entries arrive — without calling catch-up, because the sender must retry rather than discard them (see the failed-send note above).
• TestClusterReplicationFactorThreeOnFiveNodes runs 5 members at RF=3 and asserts each bucket lands on exactly 3 of them, that the other 2 hold none, that one/majority/all all succeed (write concern is measured against the replica set, not the member count), and that every node can serve a read — including the two that hold no copy.

For throughput, see bench/RESULTS.md. To reproduce a three-node cluster on one host:

# node 1 (it mints the shared cluster token)
HOARDB_DATA_DIR=/tmp/n1 HOARDB_LISTEN=127.0.0.1:4433 HOARDB_REPLICATION_FACTOR=3 \
HOARDB_SEED_NODES=127.0.0.1:4433,127.0.0.1:4434,127.0.0.1:4435 \
HOARDB_ROOT_USER=admin HOARDB_ROOT_PASSWORD=... ./hoarddb-server &

# copy /tmp/n1/cluster.token to the other nodes' data directories first
HOARDB_DATA_DIR=/tmp/n2 HOARDB_LISTEN=127.0.0.1:4434 ... ./hoarddb-server &
HOARDB_DATA_DIR=/tmp/n3 HOARDB_LISTEN=127.0.0.1:4435 ... ./hoarddb-server &

A node starts happily while its peers are still down: the ring is built from configuration, and unreachable peers are retried in the background.

Source: docs/user/replication.md in the repository.