Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Load a file: hoarddb ingest

hoarddb ingest loads a BSON file or a JSON Lines file into one bucket, and creates the bucket if it is not there. It is one command, it reports what it did, and if it is interrupted you run the same command again and it carries on.

It reads two kinds of file:

• a .bson file — BSON documents written one after another, as a BSON dump tool writes them;
• JSON Lines (.jsonl, .ndjson or .json) — one document per line, in plain JSON or Extended JSON.

It does not read a JSON array or a compressed file. A directory of .bson files loads one bucket per file (Load a directory). Each refusal names the command that fixes it (see When it says no).

Load your first file

Start a server (Getting started), then, from the directory the server runs in:

hoarddb ingest users.bson --into app.users --address 127.0.0.1:7433 --insecure

--into app.users means database app, bucket users; the text before the first dot is the database. The bucket is created as a hash bucket. To create a different store type, say so:

hoarddb ingest events.jsonl --into app.events --type fifo --address 127.0.0.1:7433 --insecure

A real run, against a fresh server (this one listens on port 27533; 5,000 documents, then 3,000 lines), followed by counting what is in the buckets. The summary line ends with the elapsed time and a rate; both depend on your machine, so this page shows them as <time> and <rate>:

$ hoarddb ingest users.bson --into app.users --address 127.0.0.1:27533 --insecure
WARNING: TLS certificate verification is disabled (--insecure)
Authenticated as admin
ingesting users.bson → app.users: 1,000 documents (19%)
ingesting users.bson → app.users: 2,000 documents (39%)
ingesting users.bson → app.users: 3,000 documents (59%)
ingesting users.bson → app.users: 4,000 documents (79%)
ingesting users.bson → app.users: 5,000 documents (100%)
ingest: users.bson → app.users — 5,000 read, 5,000 applied in <time> (<rate> docs/s)
  note: re-running this command applies the same documents to the same keys; the count will not change.

$ hoarddb ingest events.jsonl --into app.events --type fifo --address 127.0.0.1:27533 --insecure
WARNING: TLS certificate verification is disabled (--insecure)
Authenticated as admin
ingesting events.jsonl → app.events: 1,000 documents (32%)
ingesting events.jsonl → app.events: 2,000 documents (66%)
ingesting events.jsonl → app.events: 3,000 documents (100%)
ingest: events.jsonl → app.events — 3,000 read, 3,000 applied in <time> (<rate> docs/s)
  note: app.events is a fifo bucket — re-running this command appends a second copy of every document.

$ cat counts.hql
use app
db.users.count()
db.events.count()
exit
$ hoarddb -address 127.0.0.1:27533 -insecure < counts.hql
Switched to database 'app'
5000
3000

(The shell also prints its banner and the sign-in line; they are left out here.)

Progress and the summary go to standard error, so --json can have standard output to itself. The connection flags are the ones dump takes: --address, --user, --password, --auth-token, --config and --insecure.

Flags may come before or after the file name.

Load a directory

--into-db takes a directory of .bson files and loads each one into a bucket named after the file, in the database you name: users.bson becomes app.users.

hoarddb ingest ./dump/app --into-db app --address 127.0.0.1:7433 --insecure

A real run, against a fresh server, over a directory of users.bson (3,000 documents), orders.bson (2,000), events.2024.bson (500) and a users.metadata.json. The first command is a dry run; the third is the same command run again:

$ hoarddb ingest dump --into-db app --allow-dots --dry-run --address 127.0.0.1:27534 --insecure
note: the uuid in 1 metadata file(s) is ignored: a bucket's identity is its name
indexes in users.metadata.json (ingest creates none: an index can only be declared when a bucket is created):
  email_1 (unique): hoardDB can declare it
  age_-1: hoardDB cannot declare it: age is descending, and hoardDB indexes are ascending
  Ingest creates app.users without them. To have the ones hoardDB can declare, create the bucket first, then run this command again (it fills a bucket that exists):
    create bucket users in app { type: hash, indices: [{fields: ["email"], unique: true, name: "email_1"}] }
ingest (dry run): events.2024.bson → app.events.2024 — 500 read, would apply 500; it connected and read only, and nothing was written
  would create bucket app.events.2024 as hash
...
ingest (dry run): dump → app — 3 of 3 files, 5,500 read, would apply 5,500 in <time>; it connected and read only, and nothing was written

$ hoarddb ingest dump --into-db app --allow-dots --address 127.0.0.1:27534 --insecure
...
ingest: events.2024.bson → app.events.2024 — 500 read, 500 applied in <time> (<rate> docs/s)
ingest: orders.bson → app.orders — 2,000 read, 2,000 applied in <time> (<rate> docs/s)
ingest: users.bson → app.users — 3,000 read, 3,000 applied in <time> (<rate> docs/s)
ingest: dump → app — 3 of 3 files, 5,500 read, 5,500 applied in <time>

$ hoarddb ingest dump --into-db app --allow-dots --address 127.0.0.1:27534 --insecure
...
ingest: users.bson already ingested into app.users on 2026-10-23T06:30:14Z; not sent again (pass --again, or delete the "users.bson" entry of dump/.hoardingest, to load it again)
ingest: dump → app — 0 of 3 files, 0 read, 0 applied (3 already ingested) in <time>

(Lines marked ... are the connection banner, progress lines and the repeated per-file notes, left out here.)

That third run did not ask the server anything about users.bson: it skipped the file because dump/.hoardingest records it as finished and its path, size and first bytes are unchanged. So if you dropped or emptied app.users after the first run, the rerun does not notice and does not put the documents back. Load the whole directory again with:

hoarddb ingest dump --into-db app --allow-dots --again

or delete that file’s entry from dump/.hoardingest and run the same command again. --again loads every file in the directory, not just one: on a keyed bucket that re-puts each document, and on a fifo or lifo bucket it appends a second copy.

What it checks before it creates anything, each with the command that fixes it:

• No subdirectory. It reads one directory, one level. Point it at the directory that holds the files.
• At least one .bson file. Compressed files are named, with gunzip -k.
• No more than --max-buckets files (default 1,000). A node is published for thousands of buckets, not tens of thousands, so a directory of 12,000 files is refused with the number to pass if you know the node can hold them.
• No dotted bucket name without --allow-dots. orders.2024.bson would make a bucket named orders.2024; the refusal lists such names and shows the single-file command (--into app.orders.2024) that loads one on its own.

Files go in name order. The first file that stops (a refusal, Ctrl-C, a batch whose reply was lost) stops the run, and the files after it are not started. --rejects and --from-offset name one file, so a directory run refuses them; load that file on its own with --into.

Indexes in <name>.metadata.json

If users.metadata.json sits beside users.bson, ingest reads it and reports every index it defines before any bucket is created. It never creates one: an index can only be declared when a bucket is created, so a bucket ingest creates has none. For each index the report says whether hoardDB can declare it, and why not when it cannot (descending, a location (2dsphere) index, text, hashed, a nested path, sparse, partial, with a collation, an expiry index, or a bucket that is not a hash bucket). For the ones it can, the report prints the create bucket line that declares them. Run that line, then run the same command again: ingest fills a bucket that exists. For a bucket that already exists the report says an index cannot be added, and names a new bucket (users_indexed) to declare and the command that fills it. --json carries the same in indexes_reported.

The index on _id is not reported: every document is keyed by _id_ already. The uuid and the collection options in the file are ignored, and the file says so. A type of view or timeseries keeps that one file out of the run (exit code 2) and names the command that loads its .bson on its own. A metadata file that cannot be read refuses the run before it connects, because index definitions that are skipped without a word are worse than none.

Resuming a directory

A directory run keeps one resume record, <directory>/.hoardingest: a JSON object keyed by file name, each value the record a single file keeps. Run the same command again after an interruption and it does not send a file the record says was loaded in full and has not changed, carries on in the unfinished file at the byte where it stopped, and then loads the rest. Every check of a single-file resume applies to each file. A finished file is not asked of the server again: if you dropped its bucket since, pass --again or delete its entry. A <directory>/.hoardingest.lock stops two runs of one directory at once. --state moves the record; --no-resume keeps none; --dry-run writes none.

What ends up in the bucket

Every field of every document comes back with the same value and the same BSON type it had in the file: dates stay dates, object ids stay object ids, a Decimal128 stays a Decimal128. Field order is not preserved, and it differs by store type. Nothing is renamed, and nothing else is rewritten.

A JSON Lines file has no date type of its own. {"$date": "2024-05-01T00:00:00Z"} becomes a datetime; a plain string such as "2024-05-01T00:00:00Z" stays a string.

The key of each document

A hash, btree or heap bucket is keyed by a string. Ingest builds that key from the document’s _id and stores it in _id_, leaving _id as it was:

Keys are not tagged with their type, so the number 5 and the string "5" are the same key, and one overwrites the other. Ingest counts the overwrites it can see inside a batch (overwritten) and warns once when a file mixes _id types.

A fifo or lifo bucket is not keyed: documents are pushed in file order, and the last document in the file is the head of a lifo bucket. Their _id is not touched. A heap bucket is keyed like a hash bucket and every document goes in at priority 0.

When it says no

Ingest stops and tells you the fix rather than guessing. Each of these leaves the target as it was.

• A JSON array, or one pretty-printed object — jq -c '.[]' events.json > events.jsonl, then ingest events.jsonl.

• A gzip file (found by its first bytes, whatever its name) — gunzip -k users.bson.gz.

• A blob bucket — a .bson file holds documents, not payloads. Load the documents into a hash bucket, or keep the files on your own storage with one small document per file (Example: large file stores).

• A bucket of another type than the one you asked for with --type — ingest never changes an existing bucket’s type. Leave --type out to fill it as it is:

  $ hoarddb ingest users.bson --into app.events --type btree ...
  ingest: Bucket "events" in database "app" is type fifo; ingest was asked to create it as type btree. Ingest never changes an existing bucket's type. Run 'hoarddb ingest users.bson --into app.events' to fill it as a fifo, or, in the shell, run 'use app' and then 'drop bucket events' first if you meant to replace it.
  

• A bucket with an expiry index — documents older than the expiry are deleted within one sweep after they are loaded, and a string in the date field never expires. Pass --allow-expiring if that is what you want.

• A document it cannot key — a double, a Decimal128, a date, a boolean, a null, a document or an array as _id, an empty string, or an _id_ that is not a string. --id-as extjson keys such a document by its Extended JSON text instead. --on-error skip loads the rest.

• A line with more than one value, text after the value, or a repeated key — it names the line number. A missing newline between two concatenated files is the usual cause.

• A document over 33.5 MB, the most a request can carry.

• A bucket that does not exist, with --no-create.

• A bucket or database it has to create while a node joins the cluster — schema changes pause until the join settles. Ingest prints the server’s reason, which names the joining node and what the join waits on, then Nothing was ingested; re-run the same command. Run it again once nodes shows the join settled.

Creating a bucket needs the db-manager role on the database; filling one that exists needs only writer. The messages say which, and quote the grant command.

Skipping bad documents

By default ingest stops at the first document it cannot load, after loading everything before it. To load the rest and keep the bad ones:

hoarddb ingest users.bson --into app.users --on-error skip --rejects users.rejects.bson

A real run over a four-line file whose second line has a double as _id and whose fourth has an empty string:

$ hoarddb ingest mixed.jsonl --into app.mixed ...
ingest: mixed.jsonl line 2: _id is a double and hoardDB keys documents by a string. Pass --id-as extjson to key such documents by their JSON text, or --on-error=skip to import the others and list the skipped ones in a file (--rejects). Re-exporting with a string or object-id _id also works.
ingest: mixed.jsonl → app.mixed — 2 read, 1 applied in <time> (<rate> docs/s)

$ hoarddb ingest mixed.jsonl --into app.mixed --on-error skip --rejects mixed.rejects.jsonl ...
ingest: mixed.jsonl → app.mixed — 4 read, 2 applied, 2 skipped in <time> (<rate> docs/s)
  skipped: 2 document(s) (lines 2, 4); rejects in mixed.rejects.jsonl

$ cat mixed.rejects.jsonl
{"_id":3.5,"n":2}
{"_id":"","n":4}

The rejects file holds each skipped document exactly as it was in the input, so it is itself a valid input: fix the problem (or pass --id-as extjson) and ingest it. It is created with mode 0600 and never overwritten. The summary names skipped documents by position, never by content.

A unique index that refuses one document refuses the whole batch it was in. With --on-error skip, ingest sends that batch again one document at a time so that exactly the violating documents are skipped.

Interrupted? Run it again

A run writes a resume file next to the input, users.bson.hoardingest, before it sends each batch and again once the batch is confirmed. Press Ctrl-C once and the run finishes the batch in flight, records where it is and exits with code 4. Press it twice and it exits at once. Either way, running the same command again resumes.

$ hoarddb ingest users.bson --into app.bigusers --state s4.state --address 127.0.0.1:27533 --insecure   # Ctrl-C after a second
...
ingesting users.bson → app.bigusers: 86,000 documents (21%)
ingest: interrupted. Every document before byte 8,835,780 is in app.bigusers; run the same command to resume from there (resume file: s4.state).
ingest: users.bson → app.bigusers — 86,000 read, 86,000 applied in <time> (<rate> docs/s)

$ hoarddb ingest users.bson --into app.bigusers --state s4.state --address 127.0.0.1:27533 --insecure
...
ingesting users.bson → app.bigusers: 400,000 documents (100%) · <rate> docs/s
ingest: users.bson → app.bigusers — 400,000 read, 400,000 applied in <time> (<rate> docs/s)
  note: re-running this command applies the same documents to the same keys; the count will not change.

$ hoarddb -address 127.0.0.1:27533 -insecure < counts.hql     # db.bigusers.count()
400000

(Lines marked ... are the connection banner and earlier progress lines, left out here. The second run started at byte 8,835,780, and the count is exactly the 400,000 documents in the file.)

What a resume can promise depends on the store type:

When a reply is lost (a timeout, a dropped connection) ingest cannot know whether the server applied the batch, and exits with code 8. On a keyed bucket the next run simply sends it again. On a fifo or lifo bucket it refuses, and says so:

The last run stopped with a batch of 500 document(s) (bytes 1,288,490,188 to 1,290,588,340) in doubt: it may already have been applied. app.events is a fifo bucket, so re-sending it appends a second copy. ... then either pass --accept-duplicates to re-send it, or --from-offset 1290588340 to skip it.

At most that one batch can be added twice.

A resume refuses to go on when the situation it was interrupted in has changed:

• the input changed (a different length, or a different first megabyte) — it names the file and both lengths;
• the server address is different;
• the bucket’s indexes are different;
• the bucket now holds fewer entries than the interrupted run saw in it — the usual cause is a bucket dropped and recreated, and resuming would skip everything before the offset without a word.

--resume-anyway overrides the last three. --from-offset N starts at byte N of the file, which is how you resume deliberately after editing the file after the point where it stopped. Deleting the resume file starts over: on a keyed bucket that puts everything again; on a fifo or lifo bucket it appends a second copy of everything already loaded.

What the check cannot see: a bucket that was dropped, recreated and then filled by someone else to at least the number of entries the run had seen, or a different server that answers at the same address.

A run that finished is remembered. Ingesting the same file again into a fifo or lifo bucket is refused unless you pass --again.

--no-resume keeps no resume file. Reading standard input (-, with --format) keeps none either, and says so. If the resume file cannot be written, ingest refuses before it connects and names --state and --no-resume.

Two ingests of one input cannot run at once: a .lock file beside the resume file stops the second. If a run was killed outright, the lock is left behind; delete it, as the message says.

A dry run

--dry-run connects, checks the target and the file the way a real run would, reads the whole file, and writes nothing: no documents, no bucket, no resume file. It reports how many documents it would load and skip.

Machine-readable output

--json prints exactly one JSON object on standard output and keeps the text on standard error:

{
  "ok": true,
  "partial": false,
  "dry_run": false,
  "input": "users.bson",
  "format": "bson",
  "database": "app",
  "buckets": [
    {
      "bucket": "users2",
      "store_type": "hash",
      "created": true,
      "documents_read": 5000,
      "documents_applied": 5000,
      "documents_skipped": 0,
      "documents_overwritten": 0,
      "ids_minted": 0,
      "idempotent": true,
      "batch_budget_final": 2097152
    }
  ],
  "documents_read": 5000,
  "documents_applied": 5000,
  "documents_skipped": 0,
  "bytes_read": 502780,
  "elapsed_ms": 127,
  "resume_file": "",
  "in_doubt_batch": null,
  "indexes_reported": [],
  "warnings": [],
  "errors": []
}

(The object is printed on one line; this one was run with --no-resume, so resume_file is empty.)

errors lists skipped documents as {"document": 2, "reason": "_id is a double"}: the position (a line number for JSON Lines), never the content. When a run stops early, stopped_because holds the message.

Exit codes

Before this release a run that finished with skipped documents exited 2, the same number as a refusal, so a script could not tell “stopped” from “finished with skips”. It now exits 3, the number dump and restore already use for “some applied, some not”. A script that treated any non-zero exit as failure needs no change; one that matched 2 for skips needs to match 3.

Two connection changes in this release. A run with no credentials to send exits 1 (usage); it used to exit 7. And ingest no longer asks whether to trust a server it has not met, even on a terminal: it refuses. Connect once with hoarddb and confirm the fingerprint to record the server, or pass -insecure against a development server.

Sizes, and what the store commits at once

A batch is applied all-or-nothing, and the store commits only so much at once (about 20 MB by default, 5 MB on a node set to the smallest memtable, counting the index entries each document adds). Ingest sends batches of up to 2 MiB of documents, and if the store still refuses one it halves the batch and tries again, keeping the smaller size for the rest of the run (batch_budget_final). A single document whose indexed fields alone make one write too large is refused with the fix: index a smaller field, or load without the index and declare it afterwards.

What ingest does not carry over

• Indexes. Indexes can only be declared when a bucket is created. Ingest never applies index definitions. For one file, if a users.metadata.json sits beside users.bson, ingest says so and leaves it alone; for a directory it reports them (above). Create the bucket first, with the indexes you want (Indexes), then ingest into it.
• Users and roles, and blob payloads.
• Elements of nested buckets. Ingested documents carry none.
• Field order (above).

Two behaviours of the write path apply to ingested documents exactly as they do to any write, and are limits to know about:

• A document whose top-level field name contains ., [ or ], such as "Address.City", is refused: hoardDB reads those characters as a path, so no query or update could name that field. The server answers FIELD_TYPE, and ingest treats it like a duplicate: under --on-error=skip the document goes to --rejects and the rest load; under stop the run stops at that document. Nest the value ({Address: {City: …}}) or rename the field (Address_City) and load the rejects file again. A dump of another database whose documents carry such names refuses those documents the same way. A name with a dot inside a sub-document is stored, but no path can reach it.
• Keys are not tagged with their type (above).

A BSON DBPointer (a deprecated type) in the namespace $$hoardDB.now is refused, naming the field, because hoardDB reserves that namespace for now(). DBPointers in any other namespace load.

An index is on top-level fields only: an index declared on a path such as Address.City is refused when the bucket is created (Indexes).

Every flag

Source: docs/user/ingest.md in the repository.