Production Hardening¶

Recent authorization changes also protect /metrics with Admin permission or the configured scrape identity, and protect shared-reference-realm writes with Admin permission. Give monitoring its intended identity instead of making the endpoint anonymous. Native machine credentials use canonical tenant bindings; request headers cannot supply a substitute tenant.

Configure exclusive writers and machine defaults for service-owned graphs. Governed status and release-manifest writes have additional native guards; ordinary graph Write permission does not bypass them.

By default an IndentiaDB server starts fully open — anonymous access, allow-all authorization. That is convenient for development and a compliance failure for anything else. Setting IG_PRODUCTION=true turns startup into a fail-closed gate: the server refuses to boot unless the configuration meets a production security baseline.

IG_PRODUCTION accepts true/false, yes/no, on/off, or 1/0 (case-insensitive). Any other value refuses startup, so a typo can never silently switch the gate off.

The IG_PRODUCTION Boot Gate¶

With IG_PRODUCTION=true, the server validates the loaded security configuration at startup and refuses to boot unless all of the following hold:

Additional refusal conditions¶

IG_PRODUCTION=true also refuses to boot when:

• IG_ACL_BYPASS is set (admin ACL bypass), or IG_ACL_ENFORCE=false (global ACL off-switch).
• The Elasticsearch mirror is enabled without ACL enforcement (see below), or IG_ES_ALLOW_UNREPLICATED_WRITES is enabled.
• IG_CLUSTER_BOOTSTRAP_ARMED is not set explicitly to true or false (true only for the first bootstrap of a new, empty cluster).
• A standalone node has no durable RDF write-ahead log (IG_RDF_WAL_DIR); production does not acknowledge writes held only in memory.
• Cluster mode is configured with cluster.allow_plaintext, without security.mtls_enabled = true, or without security.tls_enabled = true for the HTTP API.
• Cluster mode disables join-identity checks (IG_REQUIRE_JOIN_IDENTITY=false) or the shard reconciler (IG_ALLOW_DISABLED_RECONCILER=true). See Cluster Posture.
• Cluster mode enables the Elasticsearch-compatible API without IG_ES_RDF_SOT=true. Without the replicated RDF write path, every :9200 document write would land only on the receiving node.
• SPARQL federation is enabled with an empty IG_SPARQL_FEDERATION_ALLOWLIST, or with an allowlist entry that is not an explicit HTTPS URL (wildcards and credentials in the URL are refused).
• The legacy raw SurrealDB proxy (IG_ENABLE_SURREAL_PROXY) is enabled alongside the Elasticsearch-compatible API.
• Encryption at rest is misconfigured — production implies IG_ENCRYPTION_REQUIRED unless explicitly opted out, and a Vault trust root (IG_CRYPTO_VAULT_CA_FILE) is mandatory.
• Experimental features that are not production-safe are enabled (e.g. IG_D5_SPREAD_EXPERIMENTAL).
• A dangerous operator escape hatch is set (see the next section).

There is no unauthenticated escape hatch: the production authentication gate is a hard bail, and the former IG_ALLOW_UNAUTHENTICATED flag is itself refused. For network-fenced development instances, simply leave IG_PRODUCTION unset.

Escape-Hatch Boot Audit¶

IndentiaDB carries a small number of deliberate escape hatches for recovery and migration — ignore modes that boot past corrupt state, relaxed fsync policies, SSRF-guard opt-outs. Each is safe by default, but in production a set hatch is usually a leftover from an incident, a stale manifest, or a typo.

Under IG_PRODUCTION=true the server evaluates every escape hatch in one audit, before any data, listener, or security file is touched. If anything is refused, startup fails with a single error that lists all violations at once, so you fix the manifest in one pass instead of discovering problems one restart at a time.

Values are matched generously so that intent is never missed: tokens are trimmed and case-insensitive (IG_WAL_RECOVERY=Truncate is refused even if a consumer would ignore that casing), truthy means 1/true/yes/on/y/t/enable/enabled, and explicit false means 0/false/no/off.

Refused settings¶

Configuration-file options with the same risk profile are refused by the security gate above rather than by this audit — notably OIDC allow_insecure_cluster_http, cluster.allow_plaintext, and IG_REQUIRE_JOIN_IDENTITY=false. None of them can be acknowledged.

Acknowledging a recovery boot¶

The settings marked acknowledgeable can be legitimately needed to bring a damaged production node back or to take an emergency backup. Acknowledge them explicitly by exact flag name:

IG_WAL_RECOVERY=ignore
IG_PRODUCTION_RECOVERY_ACK=IG_WAL_RECOVERY

An acknowledged hatch boots with a loud warning on the audit log target, while the rest of the production posture stays in force. The acknowledgement cannot linger: naming a flag that is not acknowledgeable, an unknown flag, or a flag that is not currently set to a refused value is itself a boot violation. Remove both the hatch and IG_PRODUCTION_RECOVERY_ACK once the recovery is done.

Warned settings¶

These settings do not block startup, but each is logged as a warning on the audit log target at every boot so it remains visible until removed:

Elasticsearch Mirror Requirements¶

When the Elasticsearch-compatible API on port 9200 is enabled, IG_PRODUCTION=true additionally requires:

The ES mirror receives the caller's identity via the X-Indentia-Sid header. Without a proxy token, that header would be spoofable by any client. When IG_ES_PROXY_TOKEN is set, X-Indentia-Sid is only honored if the request also carries a matching X-Indentia-Proxy-Token header (compared in constant time); a mismatch is rejected.

Client ──► Search proxy (authenticates user, injects X-Indentia-Sid
                          + X-Indentia-Proxy-Token)
                 │
                 ▼
         IndentiaDB :9200  (verifies proxy token, enforces ACL per SID)

Pair this with a NetworkPolicy so only the proxy can reach port 9200.

Cluster Posture¶

In cluster mode, production changes several defaults and refuses the settings that would weaken them.

Durable Raft acknowledgements¶

IG_RAFT_LOG_SYNC defaults to full under IG_PRODUCTION: every acknowledged Raft write is flushed and fdatasynced, so it survives power loss. Outside production the default remains range, which orders writes but issues no device flush. IG_RAFT_LOG_SYNC=range is refused in production and cannot be acknowledged.

Join identity¶

When IG_CLUSTER_ID is configured — always the case in production — a joining node verifies the cluster identity in the seed or leader's response. A response without a cluster UUID is rejected; a mismatching UUID is always fatal.

The environment variable takes precedence over the configuration file. Standalone nodes ignore the setting.

Tenant tags on distributed queries¶

Distributed-query workers re-check the tenant tag carried by each request against the replicated cluster state (see Authorization). IG_DQS_REQUIRE_TENANT controls how an untagged request from an older coordinator is treated; it defaults to the value of IG_PRODUCTION, so production refuses untagged requests. Set it to false only for the duration of a rolling upgrade — the boot audit records it as a warning.

Fixed topology¶

A production cluster's topology is operator-managed. The cluster-admin endpoints that would change it at runtime return 409 Conflict with error production_fixed_topology (not retryable):

Plan topology changes as a separate backup/restore or membership procedure.

Production Traffic Behaviour¶

• Strong reads by default. Production reads default to strong (linearizable) consistency. An explicit consistency parameter or X-Indentia-Consistency header with any value other than strong or linearizable is rejected with 400 invalid_request. (Outside production, reads stay local unless strong consistency is requested — see Read Consistency.)
• Readiness-gated traffic. Ordinary requests are admitted only while the node is ready, which in production includes a certified ACL index and a healthy audit sink. Otherwise requests receive 503 linearizable_read_unavailable (or 503 durable_audit_unavailable when the audit sink is the cause) with Retry-After: 1. /live, /ready, /health, /health/details, and /metrics stay available for diagnosis.
• No internal redirects. Production never redirects a client to an internal node address; a leader change during a request returns 503 leader_transition with Retry-After: 1. Retry through your stable client endpoint.

Durable Audit Log¶

IG_AUDIT_LOG_FILE enables an append-only, SHA-256 hash-chained audit log. It is required under IG_PRODUCTION and must be an absolute path on persistent storage (e.g. /data/audit/audit.log). Every audit event is appended as one JSON line and fsynced before the chain advances. Each record embeds the hash of the previous record:

hash(n) = SHA-256( hash(n-1) ‖ seq(n) ‖ canonical_json(event(n)) )

Editing, reordering, or deleting any record breaks the chain — tampering is detectable, not preventable-in-silence.

Startup¶

Before any listener binds, the server takes an exclusive lock on the file, verifies the complete existing chain, and appends and fsyncs a startup record. A symlinked or non-regular destination, a file locked by another process, or a corrupt or partially written chain fails startup; the server never silently starts a new chain over a damaged one.

Runtime failures¶

If an append, flush, fsync, or lock later fails, audit health becomes sticky-unhealthy:

Outside production, a sink failure is logged and recorded in audit health, but requests are not blocked. When IG_AUDIT_LOG_FILE is unset (development only), audit events are still emitted through the structured log (target audit).

Captured events¶

Verifying the chain¶

The chain is verified programmatically with verify_chain, which walks every record and reports the first break: an inserted/removed record (sequence gap), a tampered chain link, or a tampered record body.

Recommended Production Configuration¶

IG_PRODUCTION=true
IG_CLUSTER_ID=<stable-cluster-uuid>        # required; record it with your recovery secrets
IG_AUDIT_LOG_FILE=/data/audit/audit.log    # required; absolute path on persistent storage
IG_CLUSTER_BOOTSTRAP_ARMED=false           # required; true only for the first bootstrap of a new cluster

# Standalone nodes: durable RDF write-ahead log
IG_RDF_WAL_DIR=/data/rdf-wal

# Elasticsearch-compatible API on :9200
IG_ES_ACL_ENFORCE=true
IG_ES_PROXY_TOKEN=<shared-with-proxy>
IG_ES_RDF_SOT=true                         # required in cluster mode

Leave IG_RAFT_LOG_SYNC, IG_REQUIRE_JOIN_IDENTITY, and IG_DQS_REQUIRE_TENANT unset so the production defaults apply, and keep every escape hatch from the boot audit out of the manifest.

Pair this with a security.toml that enables authentication (OIDC, LDAP, or service tokens), sets allow_anonymous = false, acl.require_tenant_scope = true, acl.default_access = "deny_all", audit.enabled = true, keeps admin = "admin" in [authorization.roles], and leaves [authorization].default_permission at "none" (see Authorization). Cluster mode additionally needs security.mtls_enabled = true and security.tls_enabled = true. For a regulated bar, add encryption at rest with Vault Transit.

See Also¶

• Encryption at Rest — AES-256-GCM envelope encryption for WAL, checkpoints, and Raft state
• Authentication — providers, including tenant-scoped service tokens
• Authorization — role permissions, default_permission, and tenant scoping
• Triple-Level ACL — SID annotations and the fail-closed ACL index
• High Availability & Clustering — mTLS, linearizable reads, replication robustness