Scale & Sharding¶

IndentiaDB's scale roadmap targets 250 million documents / billions of triples on commodity hardware. The offline memory-mapped RDF index already reads billions of triples at a flat ~2 GiB RAM footprint; the surrounding layers — live ingest, the Elasticsearch projection, vector index builds, cross-shard queries, and result serialization — are being raised to that same bar. This page describes what ships today and how to enable it.

Bounded Ingest (max_delta_quads)¶

Live writes accumulate in an in-memory delta overlay on top of the offline index. The overlay is hard-bounded so a runaway UPDATE stream degrades into clean backpressure instead of an out-of-memory crash:

[limits]
max_delta_quads      = 5000000   # hard cap (default); 0 disables
max_delta_quads_warn = 4000000   # WARN threshold (default)

[limits.compaction]
auto_compaction_enabled = false  # opt-in: fold the overlay into the offline index
max_delta_quads_compact = 2000000
memory_fraction_bp      = 2500   # trigger at 25% of the cgroup memory limit
min_interval_secs       = 300

An update that would exceed the cap is rejected with HTTP 413 (delta_overlay_limit_exceeded) — see the API Reference. The overlay size is observable as the ig_delta_estimated_bytes metric, and with auto-compaction enabled the server folds the overlay into the offline index before the ceiling is reached.

Auto-compaction is standalone only: in cluster mode the setting is ignored with a warning, because the fold would race the Raft apply path. Clusters reclaim overlay memory with the online garbage collection below and move bulk data into an offline index with HA base-index attach.

Online overlay garbage collection¶

Two growth components of the overlay can be reclaimed online and cluster-safely. Both run inside the replicated apply path and tick on the replicated log index, so identically configured nodes make identical changes at the same point in the log. Both are off by default.

Streaming Projection Bootstrap¶

The Elasticsearch-compatible projection (:9200) is bootstrapped by streaming the corpus in bounded, hash-partitioned batches instead of materializing all quads in RAM — bootstrap memory is now independent of corpus size. Vector index cold builds (DiskIVF) are similarly data-parallel, with off-heap memory-mapped ID maps and keyset pagination, so building a 100M+ vector index no longer requires holding its ID maps on the heap.

On restart, both derived projections catch up incrementally from a persisted certified watermark instead of rebuilding — see Derived Projections.

Transparent Tenant Routing¶

With IG_TENANT_ROUTING=true, a tenant-scoped request on the normal /sparql and /update endpoints is routed to that tenant's shard — the same physical isolation as POST /cluster/tenant/{tenant}/update, without clients having to know the cluster API. The flag is off by default; with it off, request handling is unchanged.

A request is routed when all of the following hold: the node runs in cluster (multi-Raft) mode, the request's tenant resolves, and that tenant is mapped in the control plane. The tenant comes from the verified actor's tenant claim when authentication is enabled; the X-Indentia-Tenant header is honoured only when authentication is off. Routing is decided strictly after authentication and authorization.

• Reads evaluate against the tenant's shard replica through a filter that only exposes named graphs under https://id.indentia.ai/sources/<tenant>/ (plus the tenant's urn:indentia:rag:<tenant> graph). The default graph and every other tenant's graphs are invisible. A node that does not host the shard redirects to one that does; strong reads use the shard group's read barrier.
• Writes go through the shard leader, with WHERE evaluation restricted the same way and unconditional target-graph confinement: writes to the default graph, to foreign graphs, broad CLEAR/DROP, and LOAD are denied.

Distributed Queries¶

In a sharded cluster, POST /cluster/index/{index}/sparql (Admin) answers a SPARQL query over every shard of the index that the caller's tenant scope allows; POST /cluster/index/{index}/sparql-join always uses the materialize-and-join strategy. Nodes fan out over a dedicated Distributed Query Service (gRPC, dqs_port, default 7003, same mTLS as Raft).

Each query is answered by the first applicable strategy:

1. Hash-shuffle join (opt-in, IG_DISTRIBUTED_SHUFFLE_JOIN) — for two-way joins: rows are hash-repartitioned on the join key across nodes, joined locally, and gathered. Response header X-IG-Join-Strategy: shuffle.
2. Scatter-gather union (opt-in, IG_DISTRIBUTED_SCATTER_GATHER) — for union-safe queries: a SELECT whose only modifiers are a projection, optional DISTINCT/REDUCED, and optional OFFSET/LIMIT runs on every shard and is unioned centrally, with central de-duplication and limit application.
3. Materialize-and-join (always available) — every multi-shard query that is not union-safe (joins, ORDER BY, GROUP BY, ASK, CONSTRUCT, …) and every shape the shuffle cannot run correctly. Each shard's triples are gathered — remote shards through budget-bounded subplans — and the query runs once centrally. Correct for arbitrary cross-shard joins and modifiers, but its cost grows with the data in scope. Blank-node identity is not reconciled across shards. Response header X-IG-Join-Strategy: materialize.

A query is never answered by a silent union that would drop cross-shard matches. Only lexical term text crosses node boundaries — never node-local term IDs — so no cross-node vocabulary coordination is needed.

Resource budgets¶

Remote subplans are bounded. When a request does not carry its own limits, these defaults apply; zero or unparsable values are ignored, and there is no "unlimited" setting:

Every stage receives the remaining budget of the query, and remote calls carry client-side deadlines, so a stalled peer surfaces as a typed timeout instead of hanging. A retryable failure (peer unavailable, timeout) tries the next copy of the shard while budget remains.

Fail-closed on incomplete or oversized results¶

X-IG-Allow-Partial: true

Shard Replication and Failover¶

Each shard runs its own Raft group. The following [cluster] settings tune how shard copies are promoted, published as in-sync, and repaired. All defaults are behaviour-neutral.

Failed shard membership reconciliations back off exponentially (starting at 2 s, capped at 60 s); ig_shard_membership_sync_stuck reports 1 while a group has failed ten or more times in a row. A snapshot can be triggered on one shard group with POST /cluster/shard/{group_no}/snapshot, which returns 202 Accepted.

Control-Plane Durability¶

When the control-plane state (indices, tenants, shard map) is persisted to a meta WAL (IG_META_DATA_DIR), that WAL is now checkpointed and compacted: full segments are sealed as generations, a covering meta.checkpoint is written atomically with a checksum (sealed when encryption at rest is configured), and covered generations are removed. Boot loads the checkpoint and replays only the tail, so startup cost is proportional to the state plus the tail rather than the whole history. A crash in the middle of a checkpoint keeps the previous checkpoint and generations intact; a failed checkpoint is retried on the next control-plane write. An existing un-checkpointed meta WAL is adopted transparently.

The legacy standalone per-shard WAL data lane (shard-N.wal) is retired: a node refuses to boot over record-bearing legacy shard WALs until they are exported with indentiagraph shard-wal export — see Cluster Boot Checks.

HA Base-Index Attach¶

The delta overlay is the wrong home for bulk data; the offline memory-mapped index is the right one. In cluster mode, the replicated base-index lifecycle (command format v8) swaps in an offline-built index without a write freeze: every replica stages a byte-identical artifact in advance, all replicas switch at the same log index, and the writes that landed while the index was being built are replayed on top. Writes keep flowing; expect only a short latency spike during publication.

All lifecycle endpoints require Admin.

1. Begin. POST /cluster/base-index/begin {"epoch": E} — E must be greater than the attached epoch. From this log index on, every applied RDF mutation is also recorded in a replicated write journal.
2. Export. GET /admin/export/nquads?min_snapshot_index=N streams an N-Quads export of the whole dataset (all graphs, unfiltered). Record the X-IG-Export-Snapshot-Index response header. The export may be fuzzy — the journal replay makes the result correct — and a node that has not yet applied the begin entry refuses with 409.
3. Build the index offline, for example indentiagraph import --mode bulk-index --rdf-format nquads --file export.nq --index-output <dir>/base.
4. Publish the built index as an indentiagraph backup whose payload contains exactly one index, and push it to object storage with indentiagraph backup push. Its manifest_sha256 is the SHA-256 of the backup's manifest.json.
5. Prepare. POST /cluster/base-index/prepare with {"epoch": E, "uri": "s3://<bucket>/<prefix>/<backup-id>", "manifest_sha256": "…", "export_min_snapshot_index": <header value>, "total_bytes": <size>}. Every node pulls the artifact from its configured IG_S3_* object store, verifies the digest, dry-opens the index under <epoch_root>/epochs/E/, and reports back.
6. Attach. Poll GET /cluster/base-index until every voter reports a matching verified copy, then POST /cluster/base-index/attach {"epoch": E}. Attach is refused with 409 while a voter has not prepared the artifact or the leader has not staged it. POST /cluster/base-index/abort {"epoch": E} discards a failed cycle.

Operational limits:

• Bounded journal. The write journal holds at most 1,000,000 quads. At 95% of that, /update returns a retryable 503 until the cycle is attached or aborted; a journal that overflows anyway makes attach impossible and the cycle must be restarted.
• Epoch storage. Staged epochs live under [index] epoch_root (default: the directory of [index] path, otherwise <data>/index).
• Fetch failures degrade, not crash. A replica that cannot fetch or open a committed epoch keeps serving its pre-attach state and retries with backoff; the degraded status is visible in authenticated /health/details (base_index_fetch) and in GET /cluster/base-index, and does not affect readiness. Set IG_BASE_INDEX_FETCH_FATAL_AFTER_SECS to make the process exit after retrying that long instead, so the boot path re-fetches. A node that joins later and installs a post-attach snapshot fetches the epoch during the install.
• One-way. Once a BaseIndex command has committed, the data directory requires a release with command format v8 or later — see Upgrades and Command-Format Activation.

Tenant-Spread Sharding (Experimental)¶

By default a tenant's data lives on one shard (routed by tenant). Spread routing partitions a single large tenant across shards by named-graph IRI — one document stays whole on one shard; only cross-document operations go cross-shard.

What Ships When¶

See Also¶

• High Availability & Clustering — replication, linearizable reads, large-write robustness
• Vector Search — DiskIVF for 100M+ vector corpora
• Performance & Benchmarks — the igbench harness and measured speedups