Backup & Disaster Recovery¶

IndentiaDB ships a complete offline backup pipeline: consistent file-level backups with SHA-256 manifests, Ed25519-signed and optionally encrypted, replicated off-node to any S3-compatible object store. This page covers the operational workflow; see the CLI Reference for the full flag tables.

Consistency First¶

indentiagraph backup create performs a file-level copy — it does not quiesce writers or snapshot a live database itself. Copying a live, mutating data directory is not a recoverable backup, so the CLI fails closed: it refuses to run unless you either

• pass --source-is-consistent-snapshot — an operator attestation that all writers are stopped or that --source points at a storage-level point-in-time snapshot (LVM, ZFS, Ceph, CSI), or
• configure a snapshot provider (--snapshot-provider <exe> or IG_BACKUP_SNAPSHOT_PROVIDER) — an executable the CLI invokes to create and release a consistent snapshot around the copy.

indentiagraph backup create \
  --source /var/lib/indentiadb/data \
  --source-is-consistent-snapshot \
  --repository /mnt/backups/indentiadb

The attestation is recorded in the manifest (source_consistency: operator-attested-snapshot vs provider-snapshot).

Backup Format¶

<repository>/
└── <backup-id>/
    ├── manifest.json          # Ed25519-signed envelope with per-file SHA-256 checksums
    ├── manifest.sealed        # AEAD-sealed manifest copy (when encryption is enabled)
    └── payload/
        ├── index/<name>/...
        ├── table/<name>/...
        ├── graph/<name>/...
        └── path/<name>/...

• Integrity — every file's size and SHA-256 is recorded in the manifest; backup verify re-checks all of them and exits non-zero on any mismatch.
• Signing — manifests are wrapped in an Ed25519 signature envelope. Producers configure IG_BACKUP_MANIFEST_SIGNING_KEY_FILE + IG_BACKUP_MANIFEST_SIGNING_KEY_ID; consumers verify against public keys in IG_BACKUP_MANIFEST_VERIFYING_KEY_DIR. Under IG_PRODUCTION signed manifests are required.
• Encryption — with encryption at rest enabled, every stored file is sealed with AES-256-GCM in independently authenticated chunks, and manifest.sealed authenticates the complete file inventory. Restore fails closed without the key.
• Pre-upgrade backups — IG_BACKUP_MANIFEST_ALLOW_UNSIGNED_LEGACY=true opens a loud, read-only migration window to restore older unsigned backups; backup sign-manifest re-signs a verified legacy manifest in place to exit the window. The escape hatch can never create an unsigned backup.

Off-Node Replication to S3¶

backup push and backup pull replicate backups to any S3-compatible object store (AWS S3, Garage, Ceph RGW, MinIO) using Signature V4 with resumable multipart uploads — no AWS SDK or presigned URLs involved.

export IG_S3_ENDPOINT=https://s3.example.com
export IG_S3_BUCKET=indentiadb-backups
export IG_S3_ACCESS_KEY=...
export IG_S3_SECRET_KEY=...

# Replicate off-node
indentiagraph backup push --repository /mnt/backups/indentiadb --id bk-1752130000

# Fetch on another node
indentiagraph backup pull --repository /restore/backups --id bk-1752130000

Push semantics are designed for tamper-evidence and concurrency safety:

• The local backup is fully re-verified against its manifest before upload; a tampered backup is never pushed.
• Backup IDs are immutable in the bucket — the root manifest.json is committed last with a conditional write (If-None-Match: *), so concurrent pushers cannot overwrite each other and a failed push is safely retryable.
• Interrupted multipart uploads resume from persisted per-part state.
• Under IG_PRODUCTION, pushing a backup containing plaintext (unencrypted) payloads is refused — create the backup with encryption enabled first.

pull downloads into a staging directory, re-verifies every size and checksum plus the manifest, and only then publishes atomically. A backup ID that already exists locally is refused.

Restore¶

indentiagraph backup restore \
  --repository /restore/backups \
  --id bk-1752130000 \
  --destination /restore/data-new

Restore is generation-atomic: it verifies every selected file's checksum before writing anything, stages the restore, and publishes the destination with a single rename. The destination must be a new, non-existent directory — in-place --overwrite restore is refused because it cannot be atomic. After restore, validate the generation and switch the service (or PVC mount) to the new directory.

Selective restore takes component names: --index spo --graph employees.

Point-in-Time Recovery (PITR)¶

Continuous archiving streams every artifact to S3-compatible object storage the moment it becomes immutable — sealed RDF WAL segments and checkpoints on a standalone node, and persisted Raft snapshots of every Raft group in a cluster — and restores to a chosen time or WAL generation. It is off by default; with it off, behaviour is unchanged.

How archiving stays loss-free:

• Spool at seal time. When an artifact becomes immutable it is hardlinked into a spool on the same filesystem. An artifact is never removed from its primary location before its spool entry exists.
• Acknowledged, ordered uploads. One uploader works through the spool in strict FIFO order, retrying the head with exponential backoff (capped at 60 s). A spool entry is removed only after the object and its .meta.json sidecar are acknowledged, so the bucket always holds a prefix of the local history.
• Tamper evidence. WAL artifacts carry a sealed manifest inside their sidecar, so an altered sidecar cannot mislead a restore; Raft snapshots are already self-authenticating files.
• Timelines. A delta-overlay fold or a restore starts a new timeline, so WAL generation numbers never collide in the bucket. If a fold was interrupted in a way that leaves the timeline ambiguous, the archiver goes degraded: it keeps spooling but uploads nothing (ig_pitr_degraded = 1).

Objects are stored under <IG_PITR_S3_PREFIX>/<cluster-uuid>/:

wal/<timeline>/segment-g<N>.<sha256>                    (+ .meta.json)
wal/<timeline>/checkpoint-g<N>.<sha256>                 (+ .meta.json)
wal/<timeline>/timeline.json
raft/<meta|group-N>/node-<id>/<snapshot-id>.<sha256>    (+ .meta.json)

Enabling PITR¶

Requirements — the server refuses to boot with IG_PITR_ARCHIVE=true unless they hold:

• A stable, operator-configured IG_CLUSTER_ID (the bucket prefix embeds it; a generated development identity would strand the archive).
• Encryption at rest configured, or an explicit IG_PITR_ALLOW_PLAINTEXT=true.
• On a standalone node, the RDF WAL enabled (IG_RDF_WAL_DIR) — without it there is nothing durable to archive.
• The S3 connection from the IG_S3_* variables in Off-Node Replication to S3; HTTPS is required under IG_PRODUCTION.

Archive health is exported on /metrics: ig_pitr_last_acked_gen (highest WAL generation acknowledged by the bucket — a true watermark), ig_pitr_spool_bytes, ig_pitr_spool_entries, ig_pitr_upload_failures_total, ig_pitr_timeline_advances_total, and ig_pitr_degraded.

Restoring to a point in time¶

All pitr commands read the same S3 variables and default to the cluster UUID of IG_CLUSTER_ID (override with --cluster <uuid>). See the CLI Reference for every flag.

# Timelines and archived Raft groups
indentiagraph pitr list

# Standalone: download and verify the WAL chain up to a time (or --upto-gen N;
# omit both for the latest) into a fresh directory
indentiagraph pitr restore-wal \
  --dest /restore/rdf-wal \
  --upto-time 2026-09-03T02:00:00Z
# then start the server with IG_RDF_WAL_DIR=/restore/rdf-wal

# Cluster: seed each voter's Raft state for a group from an archived snapshot
# (--latest, --upto-time <t>, or --snapshot-id <id>); repeat per group and per voter
indentiagraph pitr restore-raft --group meta --node-id 1 --data-root /data --latest
indentiagraph pitr restore-raft --group 7    --node-id 1 --data-root /data --latest

# Chain-preserving retention: never removes what a retained restore point still needs
indentiagraph pitr prune --older-than 30d --dry-run

HA Backup Sets¶

A file-level backup create copies one node's data directory. For Raft clusters, a Backup Set instead captures authenticated Raft snapshots from the live leader, together with the membership and custody evidence needed to restore them — and can only be restored as a unit.

Both exports follow the same discipline:

• Only under IG_PRODUCTION, with a configured IG_CLUSTER_ID and the production custody key, and only for an Admin caller presenting the cluster-internal reconciler identity.
• An empty request body and a canonical transaction UUID. The request is pinned to the expected leader with x-indentiagraph-expected-leader-node-id and x-indentiagraph-expected-leader-term, so a lost response cannot be blindly replayed against a different leader after failover.
• Idempotent replay of the same transaction; a transaction directory is never overwritten. The receipt (attestation.json) is published last, next to the archive.
• Record the receipt's manifest SHA-256 and the archive's SHA-256 through a second channel. Every restore command requires that independently obtained manifest digest.

Incarnation fencing¶

In the production HA profile every voter process runs under a signed incarnation claim issued by the operator and bound to the cluster UUID, node ID, a monotonically increasing epoch, and the Pod UID. Once a replacement epoch is accepted, requests from the older process are rejected before they can mutate state. Backup Sets capture each voter's incarnation fence, AEAD-sealed under the custody key and bound to that exact manifest, so fences cannot be replayed across sets and a restored cluster keeps stale pre-restore processes fenced out.

Cluster Backup Set¶

Consistency. Each group's snapshot is captured on that group's live leader behind a linearizable read barrier, so it is a linearizable point for that group. Across groups the set is not a global point-in-time transaction; instead the export proves a stable window: the control-plane allocation fingerprint (index → group → copies) is recorded before the first capture and re-verified after the last, the metadata group is captured last, and the incarnation fences are collected twice and must match. Any drift aborts the export, which is safe to retry. A write that raced the capture of its shard is either fully inside that shard's snapshot or fully absent — never torn — but two writes to different shards near the capture may land on opposite sides.

Not covered:

• Unallocated standalone shard WALs. A non-empty one refuses the export unless IG_CLUSTER_BACKUP_ALLOW_UNCOVERED_WAL is set; allowed ones are listed in the receipt so the gap is visible.
• The :9200 Elasticsearch mirror store. With IG_ES_RDF_SOT=false (the default) it is an independent store — back it up or re-ingest separately. With RDF as source of truth it is re-derived from the restored graph.
• Referenced base indexes. A snapshot that references an attached base index does not embed it; the referenced base generation must still exist in object storage at restore time. Do not garbage-collect base generations that a retained set references.

Groups led by other nodes are fetched over the authenticated cluster transport and re-authenticated locally before inclusion; a remotely fetched group snapshot larger than the 64 MiB transport limit refuses the export.

Restore — whole cluster, fail-closed. There is no partial restore: a version or format mismatch, a missing group artifact, or a checksum mismatch refuses with a clear error.

1. Stop every node. Data roots must be empty — move old roots aside rather than merging.

2. Distribute the archive and a second-channel plan to every node:

   {
     "formatVersion": 1,
     "expectedClusterUuid": "<cluster uuid>",
     "expectedCustodyKeyId": "<custody key id>",
     "expectedClusterSetManifestSha256": "<64 hex>",
     "restoreEpoch": "<one fresh uuid, identical on every node>",
     "bootstrapArmed": false
   }
   

3. Inspect once:

   indentiagraph backup-set cluster-inspect \
     --archive cluster-set.igcs --expected-manifest-sha256 <pin>
   

4. Prepare every node. This prepares every Raft root the node owns (metadata and shard groups), publishing each snapshot next to its root and, for a fenced set, the node's incarnation fence. It is idempotent per restore epoch and refuses a non-empty root.

   indentiagraph backup-set cluster-prepare-one \
     --archive cluster-set.igcs --plan plan.json \
     --node-id <id> --data-root /data \
     --receipt-output /data/cluster-restore-receipt.json
   

   --as-node-id <id> prepares the roots of that attested member from a maintenance pod. Node identity is not remappable: the restored node must boot with the attested ID.

5. Verify the whole cluster with every node's receipt. Do not start any node before this prints IG_CLUSTER_BACKUP_VERIFY_RECEIPTS_OK:

   indentiagraph backup-set cluster-verify-receipts \
     --archive cluster-set.igcs --expected-manifest-sha256 <pin> \
     --receipt r1.json --receipt r2.json --receipt r3.json
   

6. Start all nodes. Each group restores from its persisted snapshot, re-elects a leader, and the reconcile loop republishes shard health from the restored control plane.

Fixed three-node Backup Set¶

The fixed three-node set restores into three stopped, empty voter roots with IDs 1, 2, and 3, pinned by independently obtained archive and manifest digests. Its restore is designed to be orchestrated as one transaction — withdrawing clients, stopping every voter, preparing each voter root, and gating startup on Pod-bound release material — and the CLI exposes each step:

The operator's IndentiaGraphBackupSet resource describes this export/restore transaction; see Operator: Backup and Restore for its current status.

Disaster Recovery Runbook¶

Losing a node or PVC entirely:

1. Routine (before disaster): backup create (consistent + sealed) → backup verify → backup push on a schedule.
2. Recover: on a fresh node, set IG_CLUSTER_ID to the original cluster UUID and restore the tenant encryption key material, then:

indentiagraph backup pull --repository /restore/backups --id <latest-id>
indentiagraph backup restore \
  --repository /restore/backups --id <latest-id> \
  --destination /restore/data

1. Validate and switch the data mount to the restored generation, then start the server.

RPO / RTO¶

• RPO equals your snapshot → create → push cadence (nightly backups ⇒ RPO ≤ 24 h). With PITR enabled, RPO drops to roughly the seal cadence (IG_PITR_SEAL_INTERVAL_SECS) plus upload lag — watch ig_pitr_last_acked_gen.
• RTO is pull time + restore time, scaling with backup size and bandwidth.

See Also¶

• CLI Reference — indentiagraph backup — full flag tables for all nine subcommands
• CLI Reference — indentiagraph backup-set and indentiagraph pitr — HA Backup Set and point-in-time restore commands
• High Availability & Clustering — take a verified backup before a command-format upgrade
• Encryption at Rest — sealed backups and key custody
• Production Hardening — fail-closed backup policies under IG_PRODUCTION