Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Is this node full?

This page has not been checked against ring mode yet. Its metric names and counts describe the catalog the node kept before ring mode. Limitations lists what ring mode does not do yet.

A node runs out of one of four things: file descriptors, memory mappings, memory, or disk. hoardDB measures all four against the limits your kernel and your filesystem actually enforce, and tells you which one is close. Read which resource is red before you act: the right action differs by resource, and for disk the usual reflex, adding a node, is the wrong one.

The one-sentence version: look at which resource is red. For fds or mappings, raise the kernel limit. For memory under load, add a node. For disk, free space on this node: a red node votes against new members and copies in no bucket it does not already hold, but in a cluster of three or more the others can still admit a member, and a new node barely helps a full disk either way.

Where to look

You do not need to set anything up to be told. Three channels carry the same readings:

• The node’s log. A WARN line is written when a resource crosses its line, and repeated at most once an hour while it stays there. Each line names the action for that resource. This is always on.

• hoardDB status. Any authenticated session shows one row per resource:

  [h]oardDB> status
  STATUS
  ---
  ...
  Healthy:      true
  Capacity:
    fds         0.00  threshold 0.80  ok
    mappings    0.00  threshold 0.80  ok
    memory      0.00  threshold 0.80  ok
    disk        0.32  threshold 0.85  ok
  

  This is always on too.

• Prometheus, at /metrics. Off by default; see Turning on the gauges.

What the states mean

Neither unavailable nor stale means healthy. A node that cannot be measured is a node you know nothing about.

A red row in status is followed by one line pointing you to the log, because the log line carries the action and, for memory, which of two things red means.

What to watch, with Prometheus

hoarddb_node_capacity_ratio{resource} is each resource’s reading divided by its live limit. hoarddb_node_capacity_threshold_ratio{resource} is where that resource turns red. The resource label takes the values fds, mappings, memory and disk, and a fifth, blob_disk, only on a node whose blob directory (storage.blob_dir) is on a different filesystem from the data directory; see A separate blob filesystem. Graph the ratios with:

max by (resource) (hoarddb_node_capacity_ratio)

Alert on three rules, not one. The first alone is silent when a resource stops being measured, because the ratio then disappears rather than reading zero:

# a resource is over its threshold
hoarddb_node_capacity_ratio > on(resource) hoarddb_node_capacity_threshold_ratio
# a resource can no longer be measured (the threshold is always published; the ratio is not)
hoarddb_node_capacity_threshold_ratio unless on(resource) hoarddb_node_capacity_ratio
# the sampler has stopped
time() - hoarddb_node_capacity_last_sample_timestamp_seconds > 60

Health probes for an orchestrator

With HOARDDB_METRICS_ENABLED=true the metrics listener (port 9090 by default) answers three probes. Point each kind of probe at the one that fits:

/health and /ready give the verdict status prints as Healthy:. A node is unhealthy when a resource in the capacity table is red, when the store should be sealed and is not (status shows the encryption problem), when a member refuses to confirm this node, or when its store cannot persist writes (the data directory is gone, or its disk is full or failing): every write is then refused with the cause, reads still work, and /ready answers 503 with the same reason. A data directory removed or replaced while the node runs is noticed within a tenth of a second, with or without writes: the files the node holds open no longer have a name, so nothing written to them could be found after a restart. That refusal does not lift on its own: restore the directory (from a backup, or leave it empty to start the node with no data) and restart the node. A resource that is only approaching its limit, a stale sample, and a node still waiting to be added do not make it unhealthy. /live does not follow the verdict on purpose: a node short of disk is not mended by a restart.

None of the three answers carries the machine’s CPU count, memory or load average. They need no login, so they say nothing about the host. The same facts are in the health reply an authenticated session receives, where the CPU count and memory are those the node may use: inside a container with a CPU or memory limit, the limit, not the whole machine.

$ curl -s -i localhost:9090/health
HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{"healthy":false,"node_id":"node-1","uptime":"3h12m0s","reasons":["capacity: disk is over its limit; run 'status' for the numbers"]}

The thresholds

The thresholds are policy lines, not measured cliffs. They are where this release chooses to warn you, not points at which hoardDB has been measured to fail. Where your node actually degrades depends on your hardware and workload, and no figure for it is published yet.

None of these is configurable except through disk_high_water_pct, which already exists for another reason.

Disk and disk_high_water_pct

server.disk_high_water_pct (HOARDDB_DISK_HIGH_WATER, default 85) is the point at which this node stops taking on data from the cluster. At or above it the node:

• copies in no bucket it does not already hold, whoever asks: its own rebalance after a membership change, another node handing a bucket over, an operator’s migrate, or another node’s replication stream creating the bucket on it. The refusal names the node, the bucket and the fix. It still learns every membership change, still serves the buckets it holds and takes writes to those it owns, and still gives copies away;
• stops receiving replication from a node whose stream it refused. Replication is ordered, so once this node refuses a bucket that a node’s stream creates, nothing more from that node reaches it, for any bucket, until it has room: its copies of that node’s buckets fall behind and count toward no write concern, and that node keeps its log for this one and logs a WARN saying why. With a replication factor of 2 or more, writes that need that copy (the default majority at factor 2 needs both) wait until the node has room or is removed. Free space and the stream resumes on its own;
• votes against a new member, and refuses to propose one. The vote is this node’s only: in a cluster of three or more the others can still admit one.

A refused node keeps what it has. When a copy it already holds is out of date and would normally be replaced by a fresh one, the node checks the disk first: at or above the mark it refuses the fresh copy and keeps the one it has, so nothing is deleted.

A bucket this node refuses stays where it is, on its other copies. If the membership change made this node the bucket’s owner, writes to that bucket are refused, naming the full disk, until the copy lands; if it made this node one of the bucket’s replicas, the bucket is a copy short until then. Either way the way out is the same: free space on this node, then run migrate <database> <bucket> on it (the rebalance also retries on its own), or take the node out of the cluster with remove-node <node> --token <token> run on another member.

The disk warning fires at the same point, from the same measurement, so the warning and the refusals always agree.

• 0 turns off only the refusals. The node no longer refuses copies, votes against joins or refuses to propose one, but the disk still warns, at 0.80: turning off the guard does not turn off being told your disk is filling. There is no amber line in this mode, because there is no refusal to warn you ahead of.
• A value above 100 turns off both the refusal and the warning. The threshold is then above anything a disk can reach.

When a resource is red

Read the disk row first. It is the one where the obvious action is wrong.

Why adding a node does not fix a full disk

Disk is red: free space on this node. Adding a node is not the fix. At the high-water mark this node votes against new members and copies in no bucket it does not already hold. In a one- or two-node cluster the vote blocks the join. In a cluster of three or more, the other nodes can still admit one, and a bucket the change places on this node then refuses writes, or is a copy short, until space is freed, so do not grow the cluster while a node’s disk is red. Either way, a new node frees little space on this one, and slowly: buckets move by hash and not by size, freed space comes back only after cleanup and compaction, and space held by values of 1 MiB or more is not returned at all in this release. Add nodes while every node’s disk is green, as planning.

A node replacing a copy it holds with a fresh one (a replica that fell behind its owner, or an owner taking a bucket over its own older copy) keeps the old copy until the new one has arrived and passed its checks, so both need room for a moment. When the two together would take a filesystem to the high-water mark, the node refuses the fresh copy before it deletes anything, keeps the one it has, and its log names the bucket and the space the new copy needs. Free space, then run migrate <database> <bucket> on that node.

The log line for a red disk says the same:

disk on this node is at 0.91 of its filesystem (high-water mark 0.85): free space on this node or grow the volume. At this mark the node votes against joins and copies in no bucket it does not already hold. In a cluster of three or more a majority can still admit a join, and a bucket the change places on this node then refuses writes, or is a copy short, until space is freed. Do not grow the cluster while this node is red

and, with disk_high_water_pct on, the early warning:

disk on this node is at 0.78, approaching the high-water mark 0.85: free space or grow the volume now. At the mark this node votes against joins and copies in no bucket it does not already hold

What the memory ratio counts

What a node’s memtable costs, by host size. A new node chooses its memtable from its memory limit at first start (see configuration). Measured with one client loading in batches:

These peaks are for one client loading in batches; many concurrent writers need more, so watch the memory ratio. With 32 concurrent writers streaming 2.4 GB of documents, a node peaked at 4.7 GB resident with no memory limit set, 2.7 GB under a 4 GiB limit and 2.2 GB under a 3 GiB limit, where the server’s Go memory limit (75% of the node’s limit, unless you set GOMEMLIMIT) made the garbage collector hold the heap down.

The memory ratio is anonymous memory, resident plus swapped (RssAnon + VmSwap from /proc/<pid>/status), over the memory limit. Page cache does not count: the kernel sheds it under pressure.

Swapped memory counts as used, so a node that is swapping reads high and can read above 1.0. Against memory.max or host RAM, above 1.0 means the node is over its hard limit and swapping. Against memory.high, it means the node is past the line where the kernel throttles it.

The limit is the smallest memory.max or memory.high on the process’s cgroup and every parent cgroup it can see, or the host’s MemTotal when none is set. The log says which one it is using, when the node starts and again whenever it changes:

memory limit 4.0 GiB from memory.max on cgroup /system.slice

memory limit 2.0 GiB from memory.high on cgroup /system.slice/hoarddb.service: this is a throttle line, not a hard limit. Past it the kernel slows this process and reclaims its memory; it does not kill it there. The hard limit is 4.0 GiB from memory.max on cgroup /system.slice

The two red lines read differently, on purpose:

memory on this node is at 0.86 of its memory limit 4.0 GiB (memory.max on cgroup /system.slice): anonymous memory, resident plus swapped, is near the point where this process swaps heavily or is killed. If a rebalance is running, wait for it. If the block or index caches were raised, lower them. If the load is traffic-driven, add a node. If the limit is below the host's memory, raise it.

memory on this node is at 0.86 of memory.high 2.0 GiB (cgroup /system.slice/hoarddb.service), the line where the kernel starts throttling this process and reclaiming its memory. The node is being slowed, not stopped: memory.high is not a hard limit, and the process is not killed there. If memory.high was set lower than you meant (MemoryHigh= in systemd), raise it. If a rebalance is running, wait for it. If the block or index caches were raised, lower them. If the load is traffic-driven, add a node.

Inside a container, set the memory limit on the container itself. A process in its own cgroup namespace, which is Docker’s default, cannot see a limit set above the container, for example only on a pod. The log then says the limit in use is the host’s MemTotal and that a limit above the container is not seen.

On a host that manages memory with cgroup v1, hoardDB does not read the v1 limit. It uses the host’s MemTotal and writes a warning at start and every hour saying so; if the process runs under a v1 memory limit, the memory ratio reads lower than it should.

Turning on the gauges

The gauges are exported only when /metrics is on. It is a deploy-time setting:

HOARDDB_METRICS_ENABLED=true HOARDDB_METRICS_LISTEN=192.0.2.5:9090 hoarddb-server

The log warnings and status work without it. When it is off, the node says so once at start:

capacity gauges are not exported (HOARDDB_METRICS_ENABLED is false); capacity warnings still go to this log and to hoardDB status

/metrics is unauthenticated. It listens on 127.0.0.1:9090 by default. To scrape it from another host, bind it to a private interface, as above, or firewall the port.

The series, all node-wide (none is per bucket, per database or per peer):

A separate blob filesystem

Blob payloads live in <data_dir>/blobs unless storage.blob_dir (HOARDDB_BLOB_DIR) points somewhere else. When that directory is on a different filesystem from the data directory, a node has two disks that can fill, and blob payloads are usually the larger one. hoardDB then reads both:

• status and the ratio series gain a blob_disk row beside disk, with the same threshold and the same amber line.
• The disk guard checks both against disk_high_water_pct. Either one above the mark makes the node refuse to copy buckets in and vote against a new member, and the refusal names the blob directory when that is the full one.
• hoarddb_node_blob_disk_used_bytes and hoarddb_node_blob_disk_size_bytes carry its two sides.

That is five more node-wide series (the blob_disk ratio, threshold and error counter, and the two byte gauges), none per bucket. On a node whose blob directory shares the data directory’s filesystem, including the default, nothing changes: one filesystem is one disk row. Which filesystems there are is read when the node starts, so a volume mounted over the blob directory while the node runs is seen at the next start.

A red blob_disk is read like a red disk: free space on that filesystem or grow that volume, and do not grow the cluster while it is red.

Reading your node’s limits

Every ratio divides by a limit you can read yourself. Replace <pid> with the server’s process id (pgrep hoarddb-server) and <data_dir> with its data directory.

# descriptors: the "Max open files" soft limit is the fds limit
grep 'Max open files' /proc/<pid>/limits
ls /proc/<pid>/fd | wc -l

# mappings
sysctl vm.max_map_count
wc -l < /proc/<pid>/maps

# disk
df -h <data_dir>

# memory: the limits on the process's cgroup and every parent it can see
cg=$(sed -n 's/^0:://p' /proc/<pid>/cgroup)
while [ -n "$cg" ]; do echo "$cg  memory.max=$(cat /sys/fs/cgroup$cg/memory.max 2>/dev/null)  memory.high=$(cat /sys/fs/cgroup$cg/memory.high 2>/dev/null)"; cg=${cg%/*}; done
grep -E 'RssAnon|VmSwap' /proc/<pid>/status

max means no limit at that level. The walk assumes cgroup v2 mounted at /sys/fs/cgroup, which is the common layout; hoardDB itself finds the mount through /proc/<pid>/mountinfo.

How many buckets a node holds

A bucket is a key range in one shared store, not a directory, a file or a database of its own. Every bucket a node serves lives in a single storage instance on that node, so a bucket costs no file, no file descriptor, no memory mapping and no goroutine of its own. Adding buckets to a node adds keys to a store that is already open.

What is left is smaller and worth knowing, because “a bucket is free” is not the claim:

• The catalog. Each bucket is a small entry in the node’s in-memory catalog. A declared index or a declared store field makes the entry larger. The catalog grows in a straight line with the bucket count and every node holds all of it, so the memory it takes is the per-bucket cost times the buckets you plan, checked against the memory of your smallest node. hoardDB publishes no per-bucket figure yet: measure it on your own hardware with your own bucket shape, and read the node’s resident memory before and after.
• Blob payloads are the exception. A blob under 64 KiB is stored inside the shared store like any other value. A larger one is a file in one node-wide payload tree, so a blob bucket costs no descriptor held open but its content still consumes disk and inodes. Do not put video, archives or large images in a blob bucket: see Large file stores, which is the pattern for that — hoardDB holds a pointer document and your storage holds the bytes.

To see how many buckets this node serves, read hoarddb_node_buckets in the metrics above, or run hoardDB status. Both are node-wide counts; hoardDB does not keep a metric per bucket, because a metric per bucket is not viable at these counts.

Not in the capacity set

• CPU is a watch signal, not a ratio: rate(process_cpu_seconds_total[5m]) against the node’s core count. Sustained, request-driven CPU is the classic case where adding a node helps. CPU during a rebalance, a reclaim or a compaction is expected, and passes.
• process_network_receive_bytes_total and process_network_transmit_bytes_total do not measure hoardDB. They read the network namespace’s counters, so on a host with shared networking they report the whole machine’s traffic. Use hoarddb_request_duration_seconds to see whether the network is slowing requests down.

Source: docs/user/capacity.md in the repository.