High Availability & Clustering¶

IndentiaDB clusters replicate through the Raft consensus algorithm: writes commit on a quorum, a failed leader is replaced by election, and lagging followers self-heal via snapshot install. This page covers the consistency guarantees clients can rely on and the operational knobs that keep replication robust under real-world load.

Read Consistency¶

Default reads: local, eventually consistent¶

By default every node answers /sparql queries from its local state. Reads are fast and remain available during partitions, but a follower (or a partitioned minority leader) can serve stale data.

Linearizable reads: consistency=strong¶

Clients that need read-your-writes / linearizable semantics opt in per request:

# Query parameter
curl "http://node:7001/sparql?consistency=strong&query=..."

# Or header (useful for POST bodies)
curl -X POST http://node:7001/sparql \
  -H "X-Indentia-Consistency: strong" \
  -H "Content-Type: application/sparql-query" \
  -d 'SELECT ...'

A strong read executes a Raft read-index barrier before the query: the node confirms it is still the quorum-acknowledged leader and waits until its state machine has applied everything committed up to the read point. This fences the classic failure mode where a stale, partitioned leader still believes it is leader and would serve outdated data.

The barrier costs one quorum round-trip plus a catch-up wait; default reads are unaffected.

Write Semantics¶

Writes are accepted on every node but are leader-authoritative: a follower does not proxy your HTTP request — it forwards the write intent (the original SPARQL UPDATE, Graph Store operation, GraphQL mutation, or Elasticsearch mutation) over the authenticated cluster transport. The leader re-authorizes the actor, evaluates the update against leader-owned state, replicates the result through Raft, and the follower returns the committed outcome to the client. No client-side redirect handling is needed for /update, GraphQL, or the ES API.

Key guarantees:

• Idempotent retries — each logical write carries a stable mutation identity bound to a semantic digest of its content. Retrying the same operation (e.g. after a timeout) returns the original committed result instead of applying twice; the same identity with different content is rejected as a conflict. The at-most-once window is five minutes — retry an ambiguous write with the same identity within it.
• Fail closed — missing leadership, an incompatible rolling-upgrade state, or an unavailable shard replica yields a bounded, retryable error. A cluster node never falls back to applying a write locally outside Raft.
• Bounded batches — Elasticsearch bulk requests are split into bounded chunks with order preserved and per-chunk results.
• Draining nodes refuse writes cleanly — a node that is shutting down answers new writes with 503 + Retry-After: 1 so clients retry elsewhere or after failover (see Graceful Shutdown and Leader Handoff).

Raft Timing Defaults¶

The default Raft timing profile is 1000 / 3000 / 5000 ms (heartbeat interval / minimum election timeout / maximum election timeout), raised from 150 / 300 / 500 ms. The new defaults apply to the server's [cluster] configuration, the sample configuration produced by indentiagraph generate-config, and the Kubernetes operator CRD.

Why: in the Raft library IndentiaDB uses, the heartbeat interval doubles as the deadline for every AppendEntries RPC, and a timed-out RPC is retried immediately. With a 150 ms heartbeat, a multi-megabyte log entry that took longer than one heartbeat to transfer was re-sent forever. A 1 s heartbeat gives every AppendEntries a 1 s budget, and 3–5 s election timeouts stop elections from firing while a follower is merely busy applying a large entry.

• Explicit settings still win. A [cluster] block that sets the timing keys overrides the new defaults. Configurations that still pin 150/300/500 keep the old, livelock-prone profile and should be raised to 1000/3000/5000.
• The server logs a startup warning when heartbeat_interval_ms is below 1000.
• heartbeat_interval_ms must be lower than election_timeout_min_ms; the configuration is rejected otherwise.
• No wire, snapshot, or on-disk format changes are involved: nodes with different timing profiles interoperate during a rolling upgrade.

Replication Robustness Under Large Writes¶

A single SPARQL UPDATE is one Raft log entry that cannot be split. A bulk write (DROP GRAPH, INSERT DATA with 100k+ triples) therefore produces a multi-megabyte entry, which historically could livelock replication: the per-RPC deadline was a single heartbeat interval, so the transfer timed out and was re-streamed forever. This is fixed by the timing profile above and by a detached, resumable send — no configuration change is needed to benefit:

While a large transfer is in flight, expect one Raft RPCError log line per heartbeat interval per target. That is the outer timeout firing on the caller, not a re-send — the detached transfer keeps streaming.

A follower that lags more than max_replication_lag entries (default 1000), or whose pending log tail exceeds IG_RAFT_SNAPSHOT_REPLAY_BYTES (see Snapshots and Restart Time), is caught up with a chunked snapshot install (3 MiB chunks) instead of individual log entries — the self-heal path that sidesteps entry size entirely.

Snapshots and Restart Time¶

The snapshot cadence bounds the log tail a restarting node must re-apply before it can serve, so it directly controls restart time. It is configured independently of max_replication_lag: previously the cadence was fixed at max_replication_lag / 2, so raising the lag threshold for replication reasons silently stretched restart replay.

Snapshots can also be triggered by hand:

Restart work itself is also cheaper: a persisted snapshot is decoded once per boot and handed out without copying, and the delta overlay is restored in bulk from the snapshot's sorted data.

Joining and Rejoining Nodes¶

join_timeout_ms ([cluster], default 30000) is the joiner's overall join budget, not a per-RPC deadline. A join proceeds in bounded rounds:

• Each join RPC declares its own wait (join_wait_ms). The leader bounds each catch-up round to at most min(join_timeout_ms / 2, 10 s, declared wait − 1 s) and answers a non-fatal join_pending instead of being cancelled mid-wait.
• The joiner's per-RPC deadline defaults to the leader's longest round plus 2 s (about 12 s), so a wedged or wrong seed costs at most ~12 s before the next seed is tried. Override it with IG_CLUSTER_JOIN_RPC_TIMEOUT_MS.
• The budget resets whenever the leader reports replication progress for the new learner; only a join that makes no progress for join_timeout_ms fails.
• Admission is idempotent: a member that is already tracked at the same address never gets a second learner entry (which would restart an in-flight snapshot transfer), and a learner is not promoted to voter before it has caught up.
• Mixed versions interoperate: an older joiner receives a short leader round inside its old client deadline, and a newer joiner against an older leader falls back to the previous retry loop.

Cluster Boot Checks¶

Cluster startup is fail-loud: misconfigurations that used to degrade silently now refuse to boot.

The standalone per-shard WAL data lane is retired. On a standalone node, POST /cluster/tenant/{tenant}/update returns 503 shard_not_allocated: tenant data writes require a clustered deployment with an allocated shard.

Upgrades and Command-Format Activation¶

Replicated Raft commands use a versioned format, and every member must be able to decode a command before the leader proposes it. The current ClusterCommand format is v9:

How activation behaves:

• Only after every member advertises support. The leader proposes a command that needs a newer format only when every voter and learner in the group advertises it. During a mixed-version rolling upgrade the new format stays inactive until the last member is upgraded; commands that require it fail closed in the meantime — a safe write refusal, never a decoding guess.
• One way per data directory. Activation is recorded as a replicated marker that survives snapshot compaction. Once a command that requires the new format has committed, binaries that predate it can no longer run on that data directory.
• Guarded publications (v9) must stay disabled on clients until every voter and learner advertises v9 and the v9 marker has committed.

Shard placement, splitting and integrity (v10 / metadata v6)¶

The distribution update retains one metadata Raft group and independent data shard groups. Routing uses the existing over-provisioned routing_num_shards; it does not use a hash ring or AP conflict resolution.

• Set cluster.node_zone or IG_NODE_ZONE to the physical failure domain. Once any zone is set, unlabelled nodes and duplicate zones are excluded. Insufficient zones leave copies unassigned. Drain before changing a hosted node's zone. This does not alter Kubernetes scheduling.
• Allocation weights estimated shard bytes and recent request rate. With auto_rebalance=true, it rechecks every 30 seconds and starts at most one improving relocation, using the existing catch-up and voter-removal protocol.
• Periodic digest pages compare native RDF and vocabulary at the same applied log position. Lag is inconclusive; confirmed disagreement makes local cluster health red. This detects divergence but does not automatically repair it or certify every derived search index.

POST /cluster/indices/{index}/split with {"num_shards": 4} starts a durable split; GET reports progress, and {"abort": true} requests abort before routing cutover. Admin is required; production also requires the configured internal reconciler identity. The new count must divide the existing routing space and be a larger multiple of the old count (factor at most 16, at most 4096 shards). Only one split runs at a time.

Hidden child groups copy bounded pages and capture concurrent changes while the source remains readable and writable. After catch-up, a short write fence allows final RDF and retry/conditional-write/blank-node state to be verified on every assigned replica. Metadata switches routing atomically; children activate and old groups remain fenced. Missing replicas prevent commit. A 30-second cutover deadline chooses abort if verification cannot finish; releasing fences still requires quorum. Client retries can be needed during cutover, activation or quorum loss.

The first implementation supports native RDF shards with canonical tenant graphs. External base artifacts and non-transferable SQL/standalone ES state refuse the split. A single tenant routing key remains on one child; splitting an index does not split one indivisible hot tenant. Keep a verified backup and perform a separate controlled v10/v6 format migration before using this feature.

Graceful Shutdown and Leader Handoff¶

On SIGTERM a node shuts down in a fixed order:

1. Write drain (always on). New writes are refused with 503 + Retry-After: 1, /ready fails with draining so load balancers stop routing new work, reads keep serving, and in-flight updates get a short, bounded wait to finish.
2. Leader handoff (opt-in). With IG_DRAIN_HANDOFF=true, for the metadata group and every shard group it leads, the node waits until a successor and a quorum are fully caught up, stops sending heartbeats, waits out its leader lease, asks the best caught-up voter to start an election, and confirms that another node now leads. A rolling restart of a leader then avoids a full election-timeout gap. Every failure degrades to an incomplete handoff and never blocks termination.
3. Ordered storage shutdown. HTTP stops, the Raft groups shut down, the Raft log storage is closed with a final durable sync, and only then do the Raft/admin and distributed-query listeners stop.

Torn-Log Recovery¶

An ungraceful shutdown or power loss can tear the tail of the Raft log. The strict default refuses to start on a corrupt entry — correct, but it can crash-loop a whole cluster when every replica tore its own tail. Several layers now prevent and repair this:

• Durable acknowledgements. With IG_RAFT_LOG_SYNC=full, the log is flushed and fdatasynced before a write is acknowledged, so acknowledged entries survive power loss.
• Torn-write detection. With IG_RAFT_LOG_FRAMING=true, each log value is written in a length- and checksum-carrying envelope, so a torn write is detected deterministically instead of being inferred from a decode failure.
• Ordered shutdown. The log storage is closed with a final sync before the process exits (see above).
• Boot-time recovery. Before its format checks run, a starting node classifies damage at the log tail. With IG_RAFT_LOG_RECOVERY=truncate it drops only entries strictly above the highest of its durable committed watermark, last applied entry, and last purged entry, then rejoins; the quorum re-replicates the missing suffix. Version skew, tampering, policy violations, and anything at or below that floor stay fail-closed. Storage-engine page corruption is never truncated: quarantine that replica's Raft data directory and let it re-seed through a snapshot install.

Write Pipelining¶

By default every SPARQL UPDATE passes through one exclusive write gate. With IG_UPDATE_PIPELINE enabled, eligible updates take the gate shared, so parsing, staging, and proposing update N+1 overlaps with the Raft apply of update N. The feature is off by default; with it off, behaviour is unchanged.

• Eligible: ground-only INSERT DATA / DELETE DATA in cluster mode. Everything else — standalone mode, temporal writes, conditional Elasticsearch writes, SHACL-validated writes, dry runs, vector-index DDL, cascading deletes, and any update with a WHERE clause, CLEAR/DROP, or blank nodes — transparently takes the exclusive path.
• Observability: shared admissions are counted in ig_update_pipeline_shared_total, runtime fallbacks in ig_update_pipeline_fallback_total.
• Isolation: each eligible update is an independent Raft entry of ground quads, so every interleaving is a valid serialization. Requests sent sequentially on one client connection keep their order, and read-after-write is unchanged. The delta overlay limit stays hard: concurrent writers reserve capacity before they stage.

Health Probes¶

The ha_ready field in /health/details is a leader-local proof: it is true only when the node itself holds quorum-confirmed leadership evidence — followers do not echo their leader's claim, and standalone nodes always report false. See the API Reference for full response shapes.

Readiness conditions¶

A /ready 503 names condition classes only — never component identities or addresses — for example {"status":"not_ready","reasons":["meta_no_current_leader","shard_apply_lagging"]}. Diagnose further with authenticated /health/details and the logs.

Codes that mirror the traffic-admission gate include index_loading, draining, audit_sink_unhealthy, acl_index_uncertified, es_runtime_failed, es_projection_uncertified, linearizable_barrier_failed, and the meta_* family (for example meta_no_current_leader, meta_quorum_unavailable, meta_local_node_catching_up).

The probe is stricter than traffic admission: with shard conditions enabled it also fails while the shard substrate is degraded, so load balancers stop routing new work to a node that cannot reconcile shards, is installing a snapshot, or hosts leaderless or lagging shard groups. These codes only affect the probe, never requests that have already arrived:

The gauges ig_reconcile_heartbeat_age_seconds and ig_snapshot_install_inflight on /metrics expose the same signals.

Metrics and Alerts¶

Every node exports its own view on /metrics. Cluster families are only present in cluster mode — on a standalone node they are absent, not zero. Per-group families report the metadata group as shard="meta", group="0" and each hosted data-shard group as shard="<index>/<n>" with its global group number, so per-group expressions should aggregate with max by (shard, group) to de-duplicate across nodes.

Recommended alert rules¶

The following Prometheus rules cover replication, cluster health, and distributed queries. Thresholds are starting points.

groups:
  - name: indentiagraph-raft
    rules:
      - alert: IGShardNoLeader
        expr: >
          sum by (shard, group) (ig_shard_is_leader) == 0
          and max by (shard, group) (ig_shard_voters) > 0
        for: 2m
        labels:
          severity: critical
      - alert: IGRaftSnapshotBacklog
        expr: max by (shard, group) (ig_shard_last_log_index - ig_raft_snapshot_last_index) > 50000
        for: 30m
        labels:
          severity: warning

The Kubernetes operator additionally generates its own release-gate alerts for the production profile — see Operator: Monitoring.

Production Checklist¶

• Odd number of nodes (3, 5, 7) — see Deployment Overview.
• security.mtls_enabled = true for the cluster transport; cluster.allow_plaintext is rejected under IG_PRODUCTION.
• Keep the default timing profile (1000/3000/5000 ms) or higher, and a generous join_timeout_ms.
• Probes: /live for liveness/startup, /ready for readiness; size startup windows from your snapshot size.
• IG_RAFT_LOG_SYNC=full (the default under IG_PRODUCTION).
• Set IG_CLUSTER_ID and consider IG_DRAIN_HANDOFF=true for rolling restarts without election gaps.
• Upgrades across a command-format bump: backup first, blue/green, no in-place downgrade.
• Monitor ha_ready (via authenticated /health/details), indentiagraph_indentiagraph_replication_lag and the per-group metrics above (via /metrics), and deploy the recommended alert rules.
• Encrypt Raft state at rest and record IG_CLUSTER_ID with your recovery secrets.