What happens when a document is indexed in IndentiaDB?

Indexing a document in IndentiaDB is turning a document into knowledge with a birth certificate. A parliamentary paper arrives from the medallion pipeline: in the bronze layer it was a raw file, in the silver layer structured data, in the gold layer a clean, validated record. But it is still a "thing in a folder" — you can retrieve it, but the graph does not yet understand anything about how it hangs together. Indexing does three things at once:

1. The document becomes a graph. The text is cut into meaningful pieces (chunks), and each piece becomes a node with edges: to the document node, to the previous chunk, to the pipeline run that produced it. The document is no longer a blob — it is a small network of related statements.

2. Every statement gets evidence and a lock. Every triple records who says it (provenance: which pipeline run, when), how certain we are (confidence, authority), how long it holds (validity), and who may read it (an ACL that sits on the triple itself, not on a folder around it).

3. It becomes findable in two ways. As a structured fact (SPARQL over triples: "which documents from committee X mention amount Y") and as text/vector (BM25 + embedding, via a derived search index). The graph is the truth; the search index is a cache pointing at it.

Write-path comparison. Where Elastic stores a document flat on _bulk and updates inverted indices + HNSW, IndentiaDB does this on POST /context/ingest: the body is embedded, an N-Triples document is generated, and that goes into the named graph in one atomic SPARQL INSERT DATA. Only then is a projection written to an ES-like index for BM25 + kNN.

Read-path comparison. In Elastic the index is the truth: what the query returns, exists. In IndentiaDB the ES index is only candidate generation — like a knn query with num_candidates first fetching candidates and then filtering. Hits are translated back to IRIs, and the RDF read path decides definitively based on the RDF-star ACLs. Search is never an endpoint but a reference into the graph.

New embedding model? In Elastic: reindex with a new mapping. In IndentiaDB: build next alongside current, validate, flip the alias. One embedding space per vector field, never mix models in one HNSW index — and the graph meanwhile simply stays correct because embeddings were never the source of truth there.

Phase 0 — Upstream (outside IndentiaDB)

Source → bronze (raw, Pulsar + S3; publish_bronze_trigger_with_properties mints the pipeline_run_id here) → parse (docling) → dedup (BLAKE2) → classify/NER (GLiNER) → silver (SQL transformations) → gold. The gold record is published on persistent://indentia/medallion/gold. The pipeline_run_id is the thread that will tie everything together (the pipeline-traceability design decision).

Phase 1 — The gold-bridge worker

The worker consumes the topic and deserializes a GoldRecord: document_id, title, source_system, modified_at, allowed_sids, chunks, tenant, domain, pipeline_run_id. gold_to_ingest() maps this to slices — the contract of the Contextual Graph:

• 1 anchor slice: kind=document, authority=0.9, body "document: {title}" — the node that represents the document as a whole.
• N chunk slices: one per GoldChunk, kind=chunk, authority=0.85.
• Chaining: every chunk gets related_to = [doc] + [previous chunk] — a star (everything → document) and a path (chunk₁ → chunk₂ → … → chunkₙ), the structure that makes multi-hop expansion possible later.
• Bitemporality: modified_at becomes both valid_from (when was this true in the world) and observed_at (when did we see it).
• ACL filtering: only allowed_sids with prefix S-* are carried over.

Phase 2 — Ingest (the context ingest handler)

POST /context/ingest with the slice batch, in order:

• Authorization — authz_guard::require(Permission::Write) plus a cross-tenant guard.
• Injection guard — every IRI segment is validated (ctx_iri::is_valid_segment), because the IRIs land literally in a SPARQL string.
• IRI minting — per slice …/contexts/tweedekamer/parl/slice/{id}; run node …/lineage/run/tweedekamer/{pipeline_run_id}; target graph …/sources/tweedekamer/parl.
• Embedding — svc.embed(&bodies): all bodies in one batch (InferenceRegistryEmbedder in the context store; env IG_CONTEXT_EMBEDDER). Before triple generation, so the vector becomes a literal on the slice.
• N-Triples generation — per slice, triple lines are built up literally (see section 8 for the complete example).
• Atomic write — everything in one INSERT DATA { GRAPH <…> { … } }. All-or-nothing: there is no state with half a document in the graph.

Phase 3 — Physical storage

The named graph lands in the triples table of SurrealDB (the triple store): SCHEMALESS, fields subject, predicate, object, graph, with a unique index on (s,p,o,g) plus idx_spo, idx_pos, idx_osp — the three permutations that make every SPARQL join direction efficient. RDF terms are serialized as objects with an _rdf_type discriminator (RDF 1.2, incl. quoted triples for RDF-star). The bitemporal variant (triple_bitemporal) adds tx_start/tx_end and valid_start/valid_end. In addition there is a second, read-optimized store: QLever-format mmap permutation indexes (indentiagraph-storage), built offline by the indentiagraph-builder component, for the standalone SPARQL engine.

Phase 4 — ES projection (the ES projection layer)

Per slice: ensure_index + index_slice → index context-tweedekamer-parl. This is the search layer: BM25 (shadow column + analyzer) and HNSW/DiskIVF for vectors. Two hard rules:

• The projection is lossy and rebuildable — the graph remains the truth.
• The ACL is re-enforced on the RDF read path; the ES hit only yields the candidate IRI.

How Elastic stores it

In Elastic the world is flat and document-centric. An index is a silo of JSON documents; the mapping determines the type per field, and a relation between two documents exists only implicitly — via a shared value in a keyword field. That relation is nowhere data: you cannot attach "since when", "according to which source" or "with what certainty" to it. Document security sits at document level. And the index is the truth.

# Elastic
PUT /parlamentaire-docs/_doc/kv-tk-2019z23964
{ "title": "Parliamentary paper 23964", "body": "Article 1. …",
  "commissie": "FVZ", "vec": [0.02, -0.05, …] }

How IndentiaDB stores the same document

The same JSON document is converted by the RDF codec into a set of triples around one subject IRI, in a named graph:

# IndentiaDB (RDF 1.2 / N-Triples)
subject : https://id.indentia.ai/sources/tweedekamer/es/parlamentaire-docs/kv-tk-2019z23964
graph   : https://id.indentia.ai/sources/tweedekamer/

<…> rdf:type           es:Document
<…> es:index           "parlamentaire-docs"
<…> es:docId           "kv-tk-2019z23964"
<…> es:tenant          "tweedekamer"
<…> es:meta/version    "7"^^xsd:integer          ← optimistic concurrency
<…> es:meta/seqNo      "421"^^xsd:integer
<…> es:field/title     "Parliamentary paper 23964"^^xsd:string
<…> es:field/body      "Article 1. …"^^xsd:string
<…> es:field/commissie "FVZ"^^xsd:string
<…> es:field/genest    "{…}"^^rdf:JSON           ← canonical JSON, sorted keys
<…> es:vector/vec      "aB3f…"^^es:vectorf32     ← base64 little-endian f32
<< <…> es:field/body "…" >> security:allowedSid "S-1-5-21-…"   ← RDF-star ACL

Physically that lands in the SCHEMALESS triples table with permutation indexes (idx_spo, idx_pos, idx_osp) — not "one row per document", but one row per statement, directly joinable in all directions.

The differences, one by one

The server reports itself as ES 9.2.4 / Lucene 10 and listens on port 9200. Kibana, Beats, the official SDKs and your own _bulk scripts work unchanged. The feature lives behind elasticsearch-compat in the ES compatibility layer, started by the CLI.

Endpoints

• Documents: GET|PUT|DELETE /{index}/_doc/{id}, POST /{index}/_doc (auto-ID), _create, _update, _source, _bulk, _mget — plus native extension /_doc/_subscribe (WebSocket).
• Search: _search, _count, _msearch, _search/scroll, _validate/query, _explain.
• Index management: PUT|DELETE|GET /{index}, _mapping, _alias(es), _settings, _stats, templates, _refresh, _open/_close, _update_by_query, _delete_by_query, _pit.
• Kibana compatibility: /_cluster/*, /_cat/*, /_nodes, /_xpack, /_license, and full /_security/* (users, roles, role_mappings, api_keys, privileges).
• Native extra: /{index}/_cooccurrence/stats (entity co-occurrence — does not exist in ES).

Write path — from _doc to triples and back

PUT /my-index/_doc/42 {"title": "…"} goes like this:

• 1. Middleware — authentication (Basic or ApiKey; Bearer/JWT not yet), privilege check at cluster/index level, tenant fence.
• 2. Leader proposal — in RDF-as-source-of-truth mode the write goes as a DocumentMutation to the Raft leader. One serialization point = no racy writes.
• 3. Codec — doc_to_triples mints the subject IRI, sets identity and OCC triples, and converts every field: scalars → typed literals (lexical form preserved: 1 vs 1.0 do not intermingle), nested objects/arrays → one canonical rdf:JSON literal, dense_vector → base64 f32 literal (es:vectorf32).
• 4. OCC validation — version or if_seq_no+if_primary_term on the leader; conflict → 409 version_conflict_engine_exception, exactly like ES. Plus a compare-and-swap digest over the codec state against mixed diffs.
• 5. Atomic commit — DELETE DATA (only old codec quads) + INSERT DATA (new ones) in one statement.
• 6. Projection — the ES document is rebuilt from the canonical RDF (triples_to_doc) and materialized in the per-index SurrealDB table: the read replica. refresh=wait_for waits until the projection has caught up with the log index.
• 7. Derived indexes — FULLTEXT shadow column (_es_ft_title) with BM25 index, HNSW per dense_vector, idx_eq_* B-trees and _es_present_* booleans on filter fields, co-occurrence tracker.

The response is pure ES again: { "_index": "…", "_id": "42", "result": "updated", "_version": 8, "_seq_no": 422, "_primary_term": 3 }.

Search path — from DSL to SurrealQL

Supported query DSL: match, match_phrase, multi_match, term(s), range, prefix, wildcard, regexp, fuzzy, exists, ids, nested, query_string, bool, dis_max, function_score, knn, plus retrievers (rrf, linear, pinned, rescorers). The SearchService picks the best execution form per query:

• match → pushdown to SurrealDB's own FULLTEXT index: _es_ft_body @N@ 'term' with search::score(N) — BM25 by SurrealDB itself, ORDER BY _score DESC. Recognized forms are accelerated by Tantivy-WAND or a native fuzzy engine.
• knn → brute-force cosine, SurrealDB native field <|k,ef|> $vec, or the persistent HNSW engine (indentiagraph-vector); filter clauses come along in the WHERE.
• query + knn together → hybrid: both run over a rank window and are fused with RRF (1/(k + rank)).
• Filters (term, terms, range, exists) → SurrealQL WHERE, served by idx_eq_* indexes and presence booleans.
• Hits are translated back to ES JSON: internal _es_*/_acl_* fields stripped, _id from the record id, _score from BM25/cosine/RRF, with hits.total{value, relation}.

ACL — the honest story

Compatibility limits

• Single-node illusion: _shards always reports 1 shard; cluster APIs are fabrications; no real shards/replicas/cross-cluster search.
• No ingest pipelines, no custom routing, version_type only internal, only painless scripts, one fixed analyzer chain (per-field analyzers are ignored), aggregations are a subset (no geo/significant_terms/composite), no percolator/more_like_this/span/suggesters/snapshots.
• retry_on_conflict must be 0 — concurrency runs via the leader, not via retries.