Elasticsearch-Compatible API¶
IndentiaDB exposes selected Elasticsearch-compatible REST endpoints on port
9200. Supported document, search, mapping, and bulk requests use familiar
Elasticsearch shapes. Compatibility does not cover the complete Elasticsearch
control plane or guarantee that every client, Kibana, or dashboard workflow works.
Test the requests your application uses against the selected IndentiaDB build.
Authentication is enforced whenever IG_ES_ACL_ENFORCE=true or IG_PRODUCTION=true. Use Basic auth (-u root:changeme), an API key header, or a trusted identity proxy — see Identity and Authentication. With both flags off (development only), the API accepts unauthenticated requests. The examples below use Basic auth.
Cluster API¶
GET / — Cluster Info¶
Returns basic cluster information.
{
"name": "indentiadb-0",
"cluster_name": "indentiadb",
"cluster_uuid": "abc123",
"version": {
"number": "8.11.0",
"build_flavor": "default"
},
"tagline": "You Know, for Search"
}
GET /_cluster/health¶
Returns cluster health status.
{
"cluster_name": "indentiadb",
"status": "green",
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 15,
"active_shards": 30,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
GET /_cat/indices¶
GET /_cat/nodes¶
Index Management¶
Templates, settings, and field capabilities¶
PUT /_index_template/{name} stores templates and applies them to new indices,
including automatic index creation. Map facet and access-control fields as
keyword before ingestion. allowed_principals is always treated as a keyword,
including with dynamic mappings.
GET /{index}/_field_caps reports field capabilities. Index settings can be read
and updated through /{index}/_settings. Index expressions can select multiple
existing indices; an expression in a mapping or settings update is not an index
name to create.
Adding a mapped filter field triggers background presence and filter backfill.
An exists query waits for the relevant presence data instead of treating an
incomplete backfill as a missing field. Changes to keyword mappings use atomic
catalog checks so concurrent updates cannot silently replace each other.
PUT /{index} — Create Index with Mapping¶
Create an index with explicit field mappings:
curl -u root:changeme \
-X PUT http://localhost:9200/articles \
-H "Content-Type: application/json" \
-d '{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"refresh_interval": "1s"
},
"mappings": {
"properties": {
"title": {
"type": "text",
"analyzer": "english"
},
"title_keyword": {
"type": "keyword"
},
"content": {
"type": "text",
"analyzer": "english"
},
"author": {
"type": "keyword"
},
"category": {
"type": "keyword"
},
"published_at": {
"type": "date",
"format": "strict_date_optional_time"
},
"view_count": {
"type": "integer"
},
"embedding": {
"type": "dense_vector",
"dims": 1536,
"index": true,
"similarity": "cosine"
}
}
}
}'
Dense Vector Field Configuration¶
The dense_vector type supports three similarity functions:
| Similarity | Description |
|---|---|
cosine |
Cosine similarity (recommended for normalized embeddings) |
l2_norm |
Euclidean distance (lower = more similar) |
dot_product |
Dot product (requires unit-length vectors) |
Full dense vector field example:
"embedding": {
"type": "dense_vector",
"dims": 768,
"index": true,
"similarity": "cosine",
"index_options": {
"type": "hnsw",
"m": 16,
"ef_construction": 100
}
}
The mapping validates dimensions, element type, similarity, and index-option
transitions. int8_hnsw stores quantized vectors in the in-memory graph and
rescores approximate candidates from source floats. int4_hnsw and bbq_hnsw
currently use int8 storage too; they do not provide native 4-bit or binary
compression. See Vector search
before selecting a format.
GET /{index} — Get Mapping¶
DELETE /{index} — Delete Index¶
Document Operations¶
POST /{index}/_doc — Index with Auto ID¶
curl -u root:changeme \
-X POST http://localhost:9200/articles/_doc \
-H "Content-Type: application/json" \
-d '{
"title": "Introduction to Graph Databases",
"content": "Graph databases store data as nodes and edges.",
"author": "alice",
"category": "databases",
"published_at": "2026-03-21T10:00:00Z",
"view_count": 0
}'
Response:
{
"_index": "articles",
"_id": "abc123xyz",
"_version": 1,
"result": "created",
"_shards": { "total": 2, "successful": 1, "failed": 0 }
}
PUT /{index}/_doc/{id} — Index with Explicit ID¶
curl -u root:changeme \
-X PUT http://localhost:9200/articles/_doc/article-001 \
-H "Content-Type: application/json" \
-d '{
"title": "SPARQL Query Language",
"content": "SPARQL is the standard query language for RDF data.",
"author": "bob",
"category": "rdf",
"published_at": "2026-03-15T09:00:00Z"
}'
GET /{index}/_doc/{id} — Get Document¶
Response:
{
"_index": "articles",
"_id": "article-001",
"_version": 1,
"found": true,
"_source": {
"title": "SPARQL Query Language",
"author": "bob"
}
}
Source id versus _id
A document body may contain its own id field with a value different from the document's _id. IndentiaDB keeps the two apart: _id addresses the document, while queries on id (term, match, sorting) address the source field, and _source in reads and search hits returns the original value unchanged (including null and structured values). Internally the source value is stored in a reserved shadow field that is never returned and cannot be set by clients.
Documents indexed before this behaviour was introduced do not carry the separate field. Re-index them from their original source — or, with IG_ES_RDF_SOT, rebuild the Elasticsearch projection from RDF. Re-indexing from the existing Elasticsearch rows cannot recover a value that was already lost.
DELETE /{index}/_doc/{id} — Delete Document¶
POST /_mget — Multi-Get¶
Fetch several documents in one request, using either a docs array or an ids array (with POST /{index}/_mget):
curl -u root:changeme \
-X POST http://localhost:9200/articles/_mget \
-H "Content-Type: application/json" \
-d '{ "ids": ["article-001", "article-002"] }'
A single request may fetch at most ES_MGET_MAX_DOCS documents (default 1000). Larger requests are rejected as a whole with the reason _mget requests at most <N> documents, got <M>; split them into smaller batches. When document-level security is enforced, documents the caller may not see are returned as "found": false.
Bulk Operations — POST /_bulk¶
The bulk API processes multiple operations in a single request. Each operation is two lines: an action line and (optionally) a document line.
curl -u root:changeme \
-X POST http://localhost:9200/_bulk \
-H "Content-Type: application/x-ndjson" \
-d '
{"index": {"_index": "articles", "_id": "1"}}
{"title": "Graph Databases", "author": "alice", "category": "databases"}
{"index": {"_index": "articles", "_id": "2"}}
{"title": "Vector Search", "author": "bob", "category": "search"}
{"update": {"_index": "articles", "_id": "1"}}
{"doc": {"view_count": 100}}
{"delete": {"_index": "articles", "_id": "old-doc"}}
'
Supported operation types:
| Operation | Description |
|---|---|
index |
Index or replace a document |
create |
Index only if the document does not exist |
update |
Partial update of an existing document |
delete |
Delete a document (no document line required) |
Writes in cluster mode and write conflicts
In a clustered (HA) deployment, _doc and _bulk writes require IG_ES_RDF_SOT=true; otherwise they are refused with 503 cluster_block_exception. See Cluster Mode. A storage-level write conflict that persists after the server's internal retries is returned as 409 version_conflict_engine_exception, which standard Elasticsearch clients treat as retryable.
Search API — GET / POST /{index}/_search¶
The _search endpoint supports a rich query DSL. All examples use POST with a JSON body.
Queries on unmapped fields
As in Elasticsearch, a leaf query on a field the index mapping does not declare matches nothing. When a match, match_phrase, match_phrase_prefix, term, terms, prefix, wildcard, range or exists query — or a bool whose must/filter contains one — targets such a field, IndentiaDB returns an empty result immediately instead of scanning the index. The shortcut is skipped for requests with aggregations or runtime_mappings, and for indices whose mapping declares no properties.
Fields that first appear on a later document are merged into the stored mapping (Elasticsearch dynamic: true behaviour); declared field types are never overwritten. With "dynamic": false or "strict", new fields are not added to the mapping — note that IndentiaDB still accepts such documents where Elasticsearch would reject them under strict.
1. Match All¶
Return all documents:
curl -u root:changeme \
-X POST http://localhost:9200/articles/_search \
-H "Content-Type: application/json" \
-d '{ "query": { "match_all": {} } }'
2. Match — Full-Text BM25¶
With AND operator (all terms must appear):
3. Match Phrase¶
All terms must appear in the specified order:
4. Multi-Match¶
Search across multiple fields simultaneously:
{
"query": {
"multi_match": {
"query": "sparql rdf knowledge graph",
"fields": ["title^3", "content", "tags"],
"type": "best_fields"
}
}
}
Multi-match types: best_fields, most_fields, cross_fields, phrase, phrase_prefix.
5. Term — Keyword Exact Match¶
6. Terms — Match Any of Multiple Values¶
7. Range¶
Numeric range:
8. Bool — Compound Queries¶
Combine must, filter, should, and must_not clauses:
{
"query": {
"bool": {
"must": [
{ "match": { "content": "sparql query" } }
],
"filter": [
{ "term": { "category": "rdf" } },
{ "range": { "published_at": { "gte": "2025-01-01" } } }
],
"should": [
{ "term": { "author": "alice" } },
{ "term": { "author": "bob" } }
],
"must_not": [
{ "term": { "status": "draft" } }
],
"minimum_should_match": 1
}
}
}
| Clause | Description |
|---|---|
must |
Required — contributes to score |
filter |
Required — does not contribute to score (cached) |
should |
Optional — boosts score; minimum_should_match controls how many must match |
must_not |
Excluded from results |
9. Exists¶
Match documents where a field has a non-null value:
10. Prefix¶
11. Wildcard¶
12. Fuzzy¶
Match terms within a given edit distance:
{
"query": {
"fuzzy": {
"title": {
"value": "grph databse",
"fuzziness": "AUTO",
"max_expansions": 50
}
}
}
}
Fuzzy expansion adds matches next to exact ones rather than replacing them. On the Tantivy text engine (see Enterprise Search), documents that match only through an edit-distance variant are returned after the exact BM25 hits and score below the weakest exact hit.
13. Nested¶
Query fields inside a nested mapping. Set ignore_unmapped: true to make the clause match nothing — instead of failing the search — on indices whose mapping does not declare the path. This lets one query carry an optional nested clause across indices with different mappings:
{
"query": {
"bool": {
"must": [
{ "match": { "content": "graph database" } }
],
"should": [
{
"nested": {
"path": "entities",
"query": { "match": { "entities.label": "IndentiaDB" } },
"ignore_unmapped": true
}
}
]
}
}
}
KNN Search — Vector / Nearest Neighbor¶
Top-level knn, kNN queries, and the Elasticsearch 8 _knn_search request
shape use the search service. After defaults, k must be at least 1 and
num_candidates must lie between k and 10,000. Source filtering, requested
fields, and highlighting apply subject to the caller's DLS restrictions.
Vector scores and scripts¶
For cosine, _score is (1 + cosine) / 2. For l2_norm, it is
1 / (1 + squared_distance). Raw distance APIs can use the opposite ordering;
do not substitute their output directly for Elasticsearch _score.
script_score supports a bounded Painless subset, including arithmetic,
comparisons, conditional expressions, local variables, params, supported
doc access, Math functions, and vector similarity functions. It ranks at
most max_result_window candidates. Loops, if statements, stored scripts,
and other languages are refused with 400. Runtime errors and negative scores
fail the request with 400 script_exception.
The intervals query supports positional text constraints. Verify the operators
used by your application; support for this query does not imply full Painless
or full Elasticsearch Query DSL compatibility.
Use the knn parameter to perform approximate nearest-neighbor search over dense_vector fields.
curl -u root:changeme \
-X POST http://localhost:9200/articles/_search \
-H "Content-Type: application/json" \
-d '{
"knn": {
"field": "embedding",
"query_vector": [0.12, 0.45, 0.78, 0.23, 0.56],
"k": 10,
"num_candidates": 100,
"filter": {
"term": { "status": "published" }
}
},
"fields": ["title", "author", "published_at"],
"_source": false
}'
| Parameter | Description |
|---|---|
field |
The dense_vector field to search |
query_vector |
The query embedding vector (must match dims) |
k |
Number of nearest neighbors to return |
num_candidates |
Number of candidate vectors to examine per shard (higher = more accurate, slower) |
filter |
Optional pre-filter applied before KNN search |
Hybrid Search — BM25 + KNN¶
Combine full-text BM25 and vector KNN scores in a single request. IndentiaDB merges the result lists using the scorer configured by ES_HYBRID_SCORER (rrf, bayesian, or linear).
curl -u root:changeme \
-X POST http://localhost:9200/articles/_search \
-H "Content-Type: application/json" \
-d '{
"query": {
"bool": {
"must": [
{ "match": { "content": "graph database knowledge" } }
],
"filter": [
{ "term": { "status": "published" } }
]
}
},
"knn": {
"field": "embedding",
"query_vector": [0.12, 0.45, 0.78, 0.23],
"k": 20,
"num_candidates": 100
},
"size": 10,
"from": 0
}'
Hybrid Scorer Configuration¶
Set the ES_HYBRID_SCORER environment variable on the server:
| Scorer | Description |
|---|---|
rrf |
Reciprocal Rank Fusion — robust default, no tuning needed |
bayesian |
Bayesian score fusion — better calibration for imbalanced corpora |
linear |
Weighted linear combination — requires manual weight tuning |
Aggregations¶
Terms Aggregation¶
{
"query": { "match_all": {} },
"aggs": {
"categories": {
"terms": {
"field": "category",
"size": 10
}
}
},
"size": 0
}
Histogram Aggregation¶
{
"aggs": {
"view_ranges": {
"histogram": {
"field": "view_count",
"interval": 100
}
}
},
"size": 0
}
Date Histogram Aggregation¶
{
"aggs": {
"articles_over_time": {
"date_histogram": {
"field": "published_at",
"calendar_interval": "month",
"format": "yyyy-MM"
}
}
},
"size": 0
}
Metric Aggregations¶
{
"aggs": {
"avg_views": { "avg": { "field": "view_count" } },
"total_views": { "sum": { "field": "view_count" } },
"max_views": { "max": { "field": "view_count" } },
"min_views": { "min": { "field": "view_count" } },
"unique_authors": { "cardinality": { "field": "author" } }
},
"size": 0
}
Nested Aggregations¶
{
"aggs": {
"by_category": {
"terms": { "field": "category" },
"aggs": {
"avg_views": { "avg": { "field": "view_count" } }
}
}
},
"size": 0
}
Combine search with SPARQL¶
An _ext block in an Elasticsearch search body is ignored. SPARQL enrichment,
knowledge-graph boosting, SPARQL filters, query expansion, and RDF output through
_ext are not implemented. Earlier versions of this documentation showed these
as working examples; do not use them as an access-control or ranking mechanism.
Enable IG_SPARQL_SEARCH_SERVICE=1 to use the implemented
SERVICE <urn:indentia:search> from SPARQL on port 7001:
PREFIX igs: <https://graph.indentia.ai/search#>
SELECT ?doc ?score ?title WHERE {
SERVICE <urn:indentia:search> {
[] igs:index "articles" ;
igs:query "knowledge graph" ;
igs:k 50 ;
igs:hit ?doc ;
igs:score ?score .
}
?doc <http://purl.org/dc/terms/title> ?title .
}
ORDER BY DESC(?score)
LIMIT 10
The service applies the caller's document-level security. It runs once with
constant parameters, then joins the top k hits to the graph pattern. Parameters
cannot depend on earlier bindings. Graph joins can remove hits, so request more
candidates than the final result limit. igs:knnField and igs:vector select
vector search; supplying text and a vector combines them with RRF.
Pagination¶
From / Size¶
Search After (Deep Pagination)¶
Use search_after with a sort value from the last result to paginate efficiently without offset overhead:
{
"query": { "match_all": {} },
"size": 10,
"sort": [
{ "published_at": "desc" },
{ "_id": "asc" }
],
"search_after": ["2026-03-01T00:00:00Z", "article-099"]
}
Scroll API¶
For bulk export, scroll retains the matching document IDs and sort keys from the
initial search. Each page reads current document values. Updates after opening
the scroll appear in later pages, and deleted documents disappear. This differs
from Elasticsearch's document snapshot at scroll creation. ES_SCROLL_MAX_HITS
bounds retained hits across open scrolls; exceeding the budget returns 429.
Scroll is refused for non-administrative callers under document-level security.
# Initialize scroll
curl -u root:changeme \
"http://localhost:9200/articles/_search?scroll=1m" \
-H "Content-Type: application/json" \
-d '{ "size": 100, "query": { "match_all": {} } }'
# Continue scrolling using the scroll_id from the previous response
curl -u root:changeme \
-X POST "http://localhost:9200/_search/scroll" \
-H "Content-Type: application/json" \
-d '{
"scroll": "1m",
"scroll_id": "<scroll_id_from_previous_response>"
}'
Field Collapsing¶
Collapse duplicate results by a field value, returning only the top-ranked document per group:
{
"query": { "match": { "content": "graph" } },
"collapse": {
"field": "author",
"inner_hits": {
"name": "other_by_author",
"size": 3,
"sort": [{ "published_at": "desc" }]
}
}
}
Highlighting¶
Return highlighted snippets showing where query terms matched:
{
"query": { "match": { "content": "graph database" } },
"highlight": {
"fields": {
"content": {
"fragment_size": 150,
"number_of_fragments": 3,
"pre_tags": ["<mark>"],
"post_tags": ["</mark>"]
}
}
}
}
Source Filtering¶
Control which fields are included in the _source of each result:
{
"query": { "match_all": {} },
"_source": {
"includes": ["title", "author", "published_at"],
"excludes": ["embedding", "content"]
}
}
Disable source entirely (useful when using fields):
Internal fields
Server-maintained fields — names starting with _es_ (storage metadata, full-text shadows, presence flags) or _acl_ (the server-stamped ACL copy) — are never included in _source, whatever the source filter. When document-level security is enforced, search requests with a filtered or disabled _source are rejected for non-administrative callers (see Document-Level Security).
Identity and Authentication¶
Request authentication on port 9200 is enforced whenever IG_ES_ACL_ENFORCE=true or IG_PRODUCTION=true. Apart from GET /health (process liveness), requests without valid credentials are rejected with 401 security_exception.
| Method | Headers | Access |
|---|---|---|
| Credentials | Authorization: Basic …, ApiKey … or Bearer … (each scheme must be enabled in the Elasticsearch security configuration) |
According to the user's roles. Superusers have full access and bypass document-level security. |
| Trusted identity proxy | X-Indentia-Proxy-Token and X-Indentia-Sid, with no Authorization header |
Read-only, bound to the tenant IG_ES_DEFAULT_TENANT |
Trusted Identity Proxy¶
A search front end that has already authenticated the end user can forward that identity instead of real credentials. A request with no Authorization header is authenticated as a proxy principal when it carries X-Indentia-Sid and an X-Indentia-Proxy-Token that matches IG_ES_PROXY_TOKEN (compared in constant time):
curl -X POST http://localhost:9200/articles/_search \
-H "X-Indentia-Proxy-Token: $IG_ES_PROXY_TOKEN" \
-H "X-Indentia-Sid: user:alice@example.com,group:finance" \
-H "Content-Type: application/json" \
-d '{ "query": { "match": { "content": "budget forecast" } } }'
- Read-only. The proxy principal holds
read,view_index_metadataandmonitoron indices andmonitoron the cluster. It is never a superuser, so document writes are refused with403. - Tenant-bound. It is bound to the tenant named by
IG_ES_DEFAULT_TENANTand only reaches indices owned by that tenant. - Fail-closed. Without a configured (non-empty)
IG_ES_PROXY_TOKEN, or with a wrong token, the request is not authenticated and gets401. A request that carries anAuthorizationheader always takes the normal credential path. - Production.
IG_PRODUCTION=truerequires bothIG_ES_ACL_ENFORCE=trueand a non-emptyIG_ES_PROXY_TOKEN; the shared secret is what makes the header identity unspoofable. See Production Hardening.
X-Indentia-Sid lists the caller's identities, comma-separated (or as repeated headers): the first value is the user, the rest are groups. Values are Windows-style SIDs (S-1-5-21-…-<rid>) or principal strings such as user:alice@example.com or group:finance. When IG_ES_PROXY_TOKEN is set, X-Indentia-Sid without the matching token is rejected with 401 security_exception. SID headers never grant administrative rights — those come only from an authenticated superuser.
Index Ownership¶
Every index is owned by a tenant, and tenant-bound callers can only use indices their tenant owns. Indices created by the RDF projection (see RDF as Source of Truth) take their owner from the tenant in the document's subject IRI. For writes without a verified tenant this is IG_ES_DEFAULT_TENANT — the tenant the proxy principal is bound to — so proxy reads reach those indices. An index without an owner is claimed by the first write that carries a tenant; an existing owner is never overwritten.
Document-Level Security¶
Set IG_ES_ACL_ENFORCE=true (exactly true or 1) to filter every read on port 9200 by the caller's identity. The server refuses to start with this flag when no ACL is configured. The global switch IG_ACL_ENFORCE=false disables ACL enforcement everywhere, including port 9200, and logs a warning at startup when combined with IG_ES_ACL_ENFORCE=true.
Declaring Visibility¶
Each document lists the principals that may read it in allowed_principals:
curl -u root:changeme \
-X PUT http://localhost:9200/articles/_doc/budget-2027 \
-H "Content-Type: application/json" \
-d '{
"title": "Budget forecast 2027",
"content": "Projected spending per department.",
"allowed_principals": ["group:finance", "user:alice@example.com"]
}'
- A missing,
nullor emptyallowed_principalsis stored as["acl:public"]: the document is visible to every caller. - Principals are compared byte-exact (case-sensitive). SIDs are matched in their full
S-1-5-21-…-<rid>form. - On every write the server stamps its own copy of the value into
_acl_allowed_principals. The search-time ACL index reads only this server-stamped copy; a_acl_allowed_principalsvalue supplied by a client is overwritten, and_acl_*fields are never returned in_source. - The caller's principals come only from the verified identity. Under enforcement, an
allowed_principalsfilter in the search request body cannot widen what the caller sees.
Upgrading existing indices
At startup, each existing index is migrated once: documents with a missing or empty ACL are stamped acl:public, and documents written before server-side stamping get their _acl_allowed_principals copy from their allowed_principals value. A persisted marker keeps the migration from running again. No manual action is required.
How Filtering Works¶
- Before candidate generation. Visibility (
acl:publicplus the caller's principals) is applied before scoring, pagination and fusion. WithES_TEXT_ENGINE=tantivyit is compiled into a per-index ACL bitset that is intersected with text, kNN and count candidates, so hidden documents never enter the top-k or the totals. As with Elasticsearch document-level security, BM25 term statistics (IDF) are still computed over the whole index. - Inside retriever fusion. Every child of a
retriever(standard,knn, and therrf,linearandbayesianfusion retrievers) and every leg ofrank.rrfreceives the same server-resolved visibility before fusion, so a hidden document cannot enter any fused result list. - Authoritative check per hit. Each returned hit is also checked against the triple-level ACL annotations in the graph (see Triple-Level ACL); hidden hits are dropped and restricted fields are removed from
_source. A document must pass both checks, so a disagreement between the document field and the graph can only hide more, never less. - Direct reads.
GET /{index}/_doc/{id},HEAD,_source,_mgetand_explainuse the same decision. A hidden document reads as not found (404/"found": false), never403.
Restrictions for Non-Administrative Callers¶
| Request | Behaviour under enforcement |
|---|---|
scroll, suggest, collapse.inner_hits |
403 security_exception |
_search with filtered or disabled _source |
403 security_exception |
retriever, rank, knn, aggs, /{index}/_count |
Served when ES_TEXT_ENGINE=tantivy is enabled; otherwise 403 security_exception |
Document writes (_doc, _create, _update, _bulk, _update_by_query, _delete_by_query) |
403 security_exception — writes require a superuser |
In search responses to non-administrative callers, hits.total is reported as a lower bound ("relation": "gte"), and aggregation results, suggestions, profile output and scroll ids are omitted. /{index}/_count on the Tantivy engine returns the exact number of visible documents.
Cluster Mode¶
In a clustered (HA) deployment every node keeps its own local Elasticsearch document store. A :9200 write that only reached the node receiving it would make the nodes silently diverge behind a load balancer. IndentiaDB therefore guards document writes (_doc, _create, _update, _bulk, _update_by_query, _delete_by_query):
| Deployment | Document writes |
|---|---|
| Standalone | Accepted |
Cluster with IG_ES_RDF_SOT=true |
Accepted and replicated through Raft |
Cluster without IG_ES_RDF_SOT |
Refused with 503 cluster_block_exception |
Cluster without IG_ES_RDF_SOT, with IG_ES_ALLOW_UNREPLICATED_WRITES=true |
Accepted into the receiving node's store only — development only |
Reads are never blocked by this guard. The refusal names both remedies:
{
"error": {
"root_cause": [{ "type": "cluster_block_exception", "reason": "unreplicated document write refused: ..." }],
"type": "cluster_block_exception",
"reason": "unreplicated document write refused: ..."
},
"status": 503
}
With IG_PRODUCTION=true, a cluster with the Elasticsearch API enabled refuses to boot unless IG_ES_RDF_SOT=true is set, and IG_ES_ALLOW_UNREPLICATED_WRITES is refused in every configuration.
RDF as Source of Truth¶
With IG_ES_RDF_SOT=true, the RDF graph is authoritative and the documents served on port 9200 are a projection of it:
- Document writes are translated into RDF and committed through the same write path as SPARQL UPDATE (Raft in cluster mode). Every node then projects the committed change into its local Elasticsearch store.
- Each document becomes the subject
https://id.indentia.ai/sources/<tenant>/es/<index>/<doc_id>. The tenant is the verified caller's tenant, orIG_ES_DEFAULT_TENANT(defaultdefault) when request security is not enforced. - While the projection is not certified against the current graph — for example during startup reconciliation — requests return
503 unavailable_shards_exceptionwithRetry-After: 1.GET /healthis exempt. - Index-management mutations that have no replicated command, such as
PUT /{index},PUT /{index}/_mappingandDELETE /{index}, are refused with503 cluster_block_exception. An index that does not exist yet is created with a dynamic mapping when the projection writes its first document.
| Variable | Default | Effect |
|---|---|---|
IG_ES_RDF_SOT |
false |
Enables RDF as source of truth. Only the exact values true or 1 enable it. |
IG_ES_WRITE_WAIT |
true |
Read-after-write: a write returns once the projection has caught up, so an immediate GET or _search sees it. Set false or 0 for latency-tolerant bulk ingest. |
IG_ES_DEFAULT_TENANT |
default |
Tenant used for writes without a verified tenant, and the tenant the proxy principal is bound to. |
IG_ES_ALLOW_UNREPLICATED_WRITES |
false |
Development-only escape hatch for node-local writes in cluster mode without IG_ES_RDF_SOT. Refused under IG_PRODUCTION. |
Read-after-write timeouts are not write failures
If the projection does not catch up in time, the write returns 503 with a reason stating that the write is already committed and will be projected asynchronously. Retry the read rather than treating the write as failed.
Limits & Error Codes¶
| Situation | Response |
|---|---|
Storage write conflict that persists after the server's internal retries (index, create, update, delete, bulk, reindex, *_by_query) |
409 version_conflict_engine_exception — retryable |
if_seq_no / if_primary_term mismatch |
409 version_conflict_engine_exception |
_mget with more than ES_MGET_MAX_DOCS documents (default 1000) |
Rejected: _mget requests at most <N> documents, got <M> |
| Leaf query on a field the mapping does not declare | 200 with an empty result, returned without scanning the index |
nested query on an undeclared path with ignore_unmapped: true |
The clause matches nothing; the rest of the query runs normally |
term, terms or range with an object or array value |
400 |
| Missing or invalid credentials (request security enforced) | 401 security_exception |
X-Indentia-Sid without a valid X-Indentia-Proxy-Token (token configured) |
401 security_exception |
| Search mode not allowed under document-level security, or document write by a non-superuser | 403 security_exception |
| Document hidden by document-level security on a direct read | 404 / "found": false |
Document write in cluster mode without IG_ES_RDF_SOT |
503 cluster_block_exception |
Projection not certified (IG_ES_RDF_SOT) |
503 unavailable_shards_exception with Retry-After: 1 |
Index-management mutation without a replicated command (IG_ES_RDF_SOT) |
503 cluster_block_exception |
Projection did not catch up after a committed write (IG_ES_WRITE_WAIT) |
503 — the write is committed; retry the read |
Filter-Index Acceleration¶
bool.filter clauses on non-text fields — term, terms, range and exists — are served from maintained indexes instead of a scan over every document.
| Variable | Default | Effect |
|---|---|---|
ES_FILTER_INDEXES |
true |
Maintains an equality/range index for each filter field. Indexes are defined on index creation, mapping updates, dynamic ingest and the RDF projection, and by a startup migration over existing indices that builds them online without a reindex. Set false, 0, no or off to disable, for example on write-heavy deployments that want to avoid per-write index maintenance. |
ES_PRESENCE_INDEXES |
true |
Also maintains per-field presence flags so exists and must_not + exists filters are index-served. Disabled by the same values; ES_FILTER_INDEXES=false turns off both. |
The filter fields are a fixed set of common security and facet fields (for example allowed_principals, source_system, document_type, language, author and labels) plus every field the mapping declares as keyword, constant_keyword, boolean, date, a numeric type, ip or version. text fields (full-text index) and dense_vector fields (vector index) are excluded.
Security Hardening¶
Everything a client sends that ends up in the storage query layer is treated as data:
- Field names (including dotted paths), sort keys and object keys from the request body are quoted or escaped. A crafted name can at worst address a field that does not exist; it cannot change the query.
- Document ids — including the values of an
idsquery — are escaped as string literals. term,termsandrangevalues must be scalars; object and array values are rejected with400.