Vector Search & RAG¶

IndentiaDB provides exact source-vector scans and approximate vector indexes, including HNSW and disk-backed DiskIVF. The guarantee depends on the query route and its exact/approximate contract. DiskIVF uses quantized postings on disk; capacity and latency must be measured for the actual vector count, dimensions, filters, replication and workload. A document may produce many chunk vectors.

Exact top-k and ANN¶

Elasticsearch-compatible kNN defaults to exact: true. It evaluates the complete allowed source-vector set, applying scope filters and server-controlled ACLs before ranking. Configuring hnsw, phnsw, diskivf or native does not allow their ANN candidate selection to restrict an exact request.

{"knn":{"field":"embedding","query_vector":[1.0,0.0],"k":10,"exact":true}}

Set exact: false explicitly to use ANN. With ES_VECTOR_SEARCH_MODE=diskivf, num_candidates controls candidate selection; the ES adapter then rescores the original source vectors and returns the best k. The default candidate budget is max(100, floor(1.5 * k)), adjusted for the requested result window and index size. More candidates/probes improve coverage at additional CPU/I/O cost. Ties use a stable record-ID order.

“DiskIVF followed by exact top 10” only means exact ranking of its candidates. It cannot recover a better source vector that ANN did not retrieve. Genuine exact top-k evaluates the complete permitted source set. A certified in-memory float index can rank that set directly when every filter and ACL restriction resolves to document IDs. Other requests scan permitted source vectors. POST /vector/topk is also a complete scan of its explicit document selection, with its own API contract: the caller must supply an already-authorized document scope.

In RDF/Raft mode (IG_ES_RDF_SOT=true), DiskIVF generations track the applied log position and repair/fault epochs. Search checks quorum and projection state before and after execution, using shard leaders. A missing required shard or an uncertifiable projection returns 503 instead of a partial result presented as complete. Independent Raft groups do not provide one atomic global snapshot.

For RAG/GraphRAG, combine authorized candidate retrieval with text relevance and graph relationships. Evaluate recall against an exact scan per embedding profile, tenant and filter selectivity. ANN and a reranker do not by themselves establish source truth or answer correctness.

What is and isn't automatic¶

It helps to be precise about what "vector search" means in IndentiaDB:

In short: IndentiaDB is vector-aware, not auto-vectorizing. The graph is gevectoriseerd in the sense that any literal or LPG property can carry a vector and be queried with native operators — but generating those vectors is a separate concern owned by the ingest pipeline.

Quantized HNSW¶

An Elasticsearch dense_vector mapping with index_options.type: "int8_hnsw" stores one signed byte per dimension plus scale metadata in the HNSW graph. The source float vectors remain in the store. Approximate searches retrieve extra candidates and rescore them from those floats.

rescore_vector.oversample controls candidate oversampling in the mapping or request. The default is 3; 0 skips source rescoring and keeps quantized scores. int4_hnsw and bbq_hnsw currently use the same int8 storage, so do not size them as native 4-bit or binary vectors. Exact queries on quantized fields read source floats because the in-memory graph no longer holds them.

DiskIVF remains a separate server mode selected by ES_VECTOR_SEARCH_MODE=diskivf. It keeps centroids in memory and quantized postings on disk. Under IG_ES_RDF_SOT, DiskIVF is the available ANN mode; a per-field HNSW mapping does not select that standalone execution path.

Elasticsearch vector _score is a similarity score where higher is better. For cosine it is (1 + cosine) / 2; for l2_norm it is 1 / (1 + squared_distance). These are not the raw distances returned by every SurrealQL or LPG vector API. Compare scores only within the same API contract.

HNSW Index¶

Resumable index generations¶

The local GenerationStore supports begin, append, checkpoint, reopen, publish, rollback, and discard for derived vector artifacts. A reopened build resumes from its durable checkpoint; uncheckpointed input must be read again. Publication checks the immutable source revision and expected vector count before changing the active generation pointer.

POST /vector/indexes/transition records the tenant's build catalog through RDF/WAL/Raft. It requires a tenant-bound Admin, a UUIDv7 mutation ID, and an expected catalog revision. Matching retries return the stored result; different input under the same mutation ID conflicts. Builders drive these transitions; the API does not create a scheduler or ingestion pipeline.

Catalog replication does not replicate ANN files. Each consumer must check its local artifact hash, embedding space, and source revision. A ready catalog entry does not certify a replica's local index. The catalog does not automatically redirect existing ES ingestion or query traffic. Keep one embedding space per physical vector field and index when rotating current and next profiles.

What HNSW Is¶

HNSW (Hierarchical Navigable Small World) is a graph-based approximate nearest-neighbour (ANN) algorithm. It builds a multi-layer proximity graph over the vector space:

• Layer 0 contains all vectors, connected to their nearest neighbours.
• Layer 1 contains a random subset (~37% of layer 0), connected more sparsely.
• Layer 2..M contain progressively sparser subsets.

Search enters at the top (sparsest) layer, greedily navigates toward the query vector, then descends to the next layer and repeats. This hierarchical structure gives logarithmic search complexity while maintaining high recall.

Layer 2:   *                *
           |                |
Layer 1:   * - * - *        * - *
           |   |   |        |   |
Layer 0:   *-*-*-*-*-*-*-*-*-*-*  (all vectors)
                    ↑
              query enters here

Key properties: - Recall depends on the corpus, filters and search effort; measure it against an exact reference. - Build time: O(N × M × log(N)) where M is the number of neighbours per node. - Memory includes stored vectors and graph edges, roughly O(N × d + N × M). - Query cost depends on graph traversal, search effort and filter selectivity; IVF uses a different centroid/probe layout.

Configuration¶

[vector]
enabled   = true
dimension = 768
metric    = "cosine"   # "cosine" | "l2" | "dot"

[vector.hnsw]
n_lists             = 100    # number of coarse cluster centroids
training_iterations = 25     # k-means training passes
default_n_probe     = 10     # clusters searched at query time
m                   = 16     # neighbours per node (build quality)
ef_construction     = 200    # candidate list size during build

DiskIVF — Disk-Backed Vector Index¶

What DiskIVF Is¶

DiskIVF is IndentiaDB's disk-backed IVF index with scalar quantization and a storage abstraction. It is not a port of Elasticsearch DiskBBQ or the DiskHIVF research system. Source-vector rescoring belongs to the ES adapter; the generic engine returns approximate results.

This makes it the right choice for corpora that exceed what fits comfortably in memory: enterprise document archives, long-horizon conversation histories, multi-tenant platforms, or any corpus in the tens-to-hundreds of millions of vectors.

Persistent vs. DiskIVF — When to Use Which¶

Architecture¶

DiskIVF uses a three-tier layout:

┌──────────────────────────────────────────────┐
│  Centroids  (RAM)                             │
│  k-means cluster centers — always in memory   │
│  ≈100 centroids × dimension × 4 bytes        │
└──────────────────┬───────────────────────────┘
                   │  score all centroids → pick top n_probe
┌──────────────────▼───────────────────────────┐
│  LRU Cache  (RAM)                             │
│  Cached posting pages or small inline lists  │
│  configurable via CACHE_SIZE                  │
└──────────────────┬───────────────────────────┘
                   │  cache miss → fetch from storage
┌──────────────────▼───────────────────────────┐
│  Posting Lists  (Disk / SurrealKV / TiKV)    │
│  Scalar-quantized vectors: 1 / 2 / 4 bits    │
│  Written via InvertedListStore abstraction    │
└──────────────────────────────────────────────┘

Search uses asymmetric distance scoring: the query vector stays at full f32 precision while only stored vectors are quantized. This gives materially better recall than symmetric quantization (where the query is also compressed).

Enabling DiskIVF¶

Add ENGINE diskivf to the native CREATE VECTOR INDEX DDL statement. This is distinct from the ES adapter setting ES_VECTOR_SEARCH_MODE=diskivf and its exact flag. SurrealQL HNSW operators and other query routes retain their own index contracts; a setting for one route does not reconfigure every vector API.

Minimal:

CREATE VECTOR INDEX doc_embeddings
  ON <http://example.org/Document>
  FIELD <http://example.org/embedding>
  METRIC cosine
  DIMENSION 1536
  LISTS 100
  ENGINE diskivf;

Fully tuned:

CREATE VECTOR INDEX doc_embeddings
  ON <http://example.org/Document>
  FIELD <http://example.org/embedding>
  METRIC cosine
  DIMENSION 1536
  LISTS 100
  ENGINE diskivf
  CACHE_SIZE 64
  QUANTIZATION_BITS 2
  SOAR true
  SOAR_THRESHOLD 1.3;

Configuration Parameters¶

The native DDL and ES adapter have different defaults. Set quantization explicitly when comparing routes or rebuilding an existing index.

Quantization Trade-offs¶

These numbers exclude IDs, norms, metadata, source vectors, duplicate assignments and storage overhead. They do not predict total disk usage or recall.

Capacity Planning¶

The ES cold-build path streams two keyset scans: reservoir training and ingest. Training retains at most max(4096, 64 * n_lists) vectors, capped at 64 MiB of raw float32 values. Source batches are bounded to 4,096 rows and 8 MiB of raw vector values. Only one cold DiskIVF build runs per process. These are component budgets, not a total process-memory ceiling.

New posting pages contain at most 1,024 vectors and 1 MiB of serialized data. IG_DISKIVF_READ_CONCURRENCY selects 1–4 concurrent page reads (default 4). IG_DISKIVF_MMAP_DIR enables disk-backed ID mapping; without it, ID mappings remain in RAM. Candidate buffers are bounded; scope/ACL checks run in batches of at most 256 IDs. ES_KNN_SCAN_MAX_CANDIDATES defaults to 200,000.

Page publication uses an atomic manifest replacement after all new pages exist. Incomplete builds and invalid/missing pages fail validation. Existing readers can retain their generation. Legacy blobs remain readable but require a full read on their first migration; rebuild into a fresh generation for consistently bounded page reads. Upgrade all readers before writing the new format.

Mutations still copy the affected posting list page by page, and old published generations need offline retirement. Production sizing for hundreds of millions of documents requires sustained ingestion, recall, peak memory, scratch-space, cold/warm p95/p99 and recovery measurements. Current tests do not establish that capacity. On the fixed 1,000-vector, 64-dimensional, 20-query fixture, candidate expansion/source rescoring improved recall@10 from 65.5% to 100% with 100 candidates; this does not generalize to all corpora.

Rust API for Vector Operations¶

use indentiagraph_vector::{
    VectorIndexConfig, SimilarityMetric, VectorIndexSearchConfig,
    PersistentAnnEngine, AnnEngine,
};

// 1. Configure the index
let config = VectorIndexConfig {
    dimension:            768,
    metric:               SimilarityMetric::Cosine,
    n_lists:              100,
    training_iterations:  25,
    default_n_probe:      10,
    ..Default::default()
};

// 2. Create and build the engine
let mut engine = PersistentAnnEngine::new();
engine.build(config).unwrap();

// 3. Train on a representative sample
//    training_vectors: Vec<Vec<f32>>, at least n_lists; use a representative sample
engine.train(training_vectors).unwrap();

// 4. Insert (upsert) documents
engine.upsert("doc1".to_string(), vec![0.12f32, -0.45, 0.88, /* ... 768 total */]).unwrap();
engine.upsert("doc2".to_string(), vec![0.03f32,  0.71, 0.22, /* ... 768 total */]).unwrap();

// 5. Search
let query_vec = vec![0.11f32, -0.42, 0.91, /* ... 768 total */];

let results = engine.search(
    &query_vec,
    &VectorIndexSearchConfig {
        k:       10,
        n_probe: Some(20),  // override default_n_probe for this query
        offset:  0,
    },
).unwrap();

for result in &results {
    println!("id={} score={:.4}", result.id, result.score);
}

// 6. Delete a vector
engine.delete("doc1").unwrap();

// 7. Persist to disk (called automatically on graceful shutdown)
engine.flush().unwrap();

The PersistentAnnEngine serializes the HNSW graph to the configured storage backend (SurrealKV or TiKV). On restart, the index is loaded from storage — no reindexing required.

Vector Search via SurrealQL¶

When documents are stored with an embedding field, use the built-in HNSW operator <|k,ef|> for nearest-neighbour queries:

-- Find 10 nearest chunks to the query embedding
SELECT id, content, source_url,
       vector::similarity::cosine(embedding, $query) AS score
FROM chunks
WHERE embedding <|10, 40|> $query
ORDER BY score DESC
LIMIT 10;

The <|k, ef|> syntax is SurrealQL's HNSW operator where k is the number of results and ef is the dynamic candidate list size (higher = better recall).

With metadata filtering (pre-filter):

-- Only search chunks from documents published after 2023
SELECT id, content, doc_id,
       vector::similarity::cosine(embedding, $query) AS score
FROM chunks
WHERE doc_published > "2023-01-01"
  AND embedding <|10, 40|> $query
ORDER BY score DESC
LIMIT 10;

Vector Search via SPARQL¶

IndentiaDB exposes vector similarity as native SPARQL filter functions under the namespace <http://graph.indentia.ai/vector/functions#> (conventionally bound to the vec: prefix).

Available functions¶

The approxNear* variants are pushed down into the vector index (HNSW or IVF depending on the configured backend) and return only the top-k candidates. The plain similarity functions evaluate per-binding without any index lookup — use them only when the surrounding pattern has already been narrowed to a small candidate set.

Example query¶

PREFIX vec:  <http://graph.indentia.ai/vector/functions#>
PREFIX ex:   <http://example.org/>

SELECT ?doc ?score WHERE {
    ?doc ex:embedding ?vec .
    BIND(vec:approxNearCosine(?vec, $query_vector) AS ?score)
    FILTER(?score > 0.7)
}
ORDER BY DESC(?score)
LIMIT 10

The $query_vector parameter is passed as a JSON-array literal in the SPARQL request:

curl -X POST http://localhost:7001/sparql \
  -H "Content-Type: application/sparql-query" \
  -H "Accept: application/sparql-results+json" \
  -d 'PREFIX vec: <http://graph.indentia.ai/vector/functions#>
      PREFIX ex:  <http://example.org/>
      SELECT ?doc ?score WHERE {
          ?doc ex:embedding ?vec .
          BIND(vec:approxNearCosine(?vec, "[0.12, -0.45, 0.88]") AS ?score)
          FILTER(?score > 0.7)
      }
      ORDER BY DESC(?score) LIMIT 10'

Vector index DDL via SPARQL¶

Vector indexes are created and managed through SPARQL extension statements. This keeps the vector layer in lockstep with the RDF data model — no separate admin API or external orchestration is needed.

-- Create an index on a class + property
CREATE VECTOR INDEX docs_emb
  ON ex:Document
  FIELD ex:embedding
  METRIC cosine
  DIMENSION 768
  LISTS 100;

-- Inspect indexes
SHOW VECTOR INDEXES;

-- Rebuild after a backfill or metric change
REBUILD VECTOR INDEX docs_emb;

-- Remove an index (data triples remain untouched)
DROP VECTOR INDEX docs_emb;

METRIC accepts cosine, l2, or inner_product. DIMENSION must match the embedding-model output exactly. LISTS controls IVF coarse-cluster count (rule of thumb: sqrt(N)). Add ENGINE diskivf to switch from in-memory to disk-backed storage — see DiskIVF for the full parameter reference and capacity planning guide.

Hybrid Text + Vector Search in SPARQL¶

Combine vec:approxNearCosine with ql:contains-word for documents that match both a keyword condition and are semantically similar to the query embedding:

PREFIX vec:  <http://graph.indentia.ai/vector/functions#>
PREFIX ql:   <http://qlever.cs.uni-freiburg.de/builtin-functions/>
PREFIX ex:   <http://example.org/>

SELECT ?doc ?score ?textScore WHERE {
    ?doc ex:embedding ?vec ;
         ex:text      ?text .
    BIND(vec:approxNearCosine(?vec, $query_vector) AS ?score)
    BIND(ql:contains-word(?text, "machine learning") AS ?textScore)
    FILTER(?score    > 0.6)
    FILTER(?textScore > 0)
}
ORDER BY DESC(?score + ?textScore)
LIMIT 10

This query requires both conditions to be met — a document must contain the phrase "machine learning" AND be semantically close to the query vector. The combined ranking ?score + ?textScore leverages both signals.

Stored Values with Vector Filtering¶

IndentiaDB supports storing arbitrary metadata alongside each vector in the HNSW index. This enables efficient post-retrieval filtering without re-querying the main store for common metadata.

use indentiagraph_vector::{VectorWithPayload, StoredValue};

// Store vector with metadata payload
engine.upsert_with_payload(
    "doc42".to_string(),
    embedding_vec,
    vec![
        StoredValue::String("category".into(), "research".into()),
        StoredValue::Integer("year".into(), 2024),
        StoredValue::Float("citation_count".into(), 157.0),
    ],
).unwrap();

// Search with payload filter
let results = engine.search_with_filter(
    &query_vec,
    &VectorIndexSearchConfig { k: 20, n_probe: Some(30), offset: 0 },
    |payload| {
        payload.get_str("category") == Some("research")
            && payload.get_int("year").map(|y| y >= 2022).unwrap_or(false)
    },
).unwrap();

Common stored values: document category, language, tenant ID, access level, publication date. Using stored values avoids a round-trip to the main key-value store for each of the k results.

RAG Pipeline (Complete Example)¶

This section shows a complete, runnable RAG pipeline: from raw documents to answering questions with an LLM.

Step 1: Chunk Documents¶

import re
from typing import List

def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> List[str]:
    """Split text into overlapping chunks of approximately chunk_size tokens."""
    # Approximate tokenization by splitting on whitespace
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk_words = words[i : i + chunk_size]
        chunks.append(" ".join(chunk_words))
        i += chunk_size - overlap
    return chunks

# Load and chunk your document
with open("my_document.txt") as f:
    raw_text = f.read()

chunks = chunk_text(raw_text, chunk_size=256, overlap=32)
print(f"Created {len(chunks)} chunks")

Step 2: Generate Embeddings¶

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dimensional

embeddings = model.encode(
    chunks,
    batch_size=64,
    show_progress_bar=True,
    normalize_embeddings=True,  # L2-normalize for cosine similarity
)

print(f"Embedding shape: {embeddings.shape}")  # (n_chunks, 384)

Step 3: Index in IndentiaDB¶

import requests
import json

INDENTIA_URL = "http://localhost:7001"

# Create SurrealQL records with embeddings
for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
    record = {
        "id":         f"chunk:{i}",
        "content":    chunk,
        "source":     "my_document.txt",
        "chunk_idx":  i,
        "embedding":  embedding.tolist(),
    }

    response = requests.post(
        f"{INDENTIA_URL}/sql",
        headers={"Content-Type": "application/json", "Accept": "application/json"},
        json={"query": f"""
            CREATE chunks CONTENT {{
                content:   $content,
                source:    $source,
                chunk_idx: $chunk_idx,
                embedding: $embedding
            }}
        """, "vars": {
            "content":   chunk,
            "source":    "my_document.txt",
            "chunk_idx": i,
            "embedding": embedding.tolist(),
        }}
    )
    response.raise_for_status()

print(f"Indexed {len(chunks)} chunks")

Step 4: Retrieve Relevant Chunks¶

def retrieve_chunks(query: str, k: int = 5) -> list[dict]:
    """Retrieve the k most relevant chunks for a query."""
    query_embedding = model.encode([query], normalize_embeddings=True)[0]

    response = requests.post(
        f"{INDENTIA_URL}/sql",
        headers={"Content-Type": "application/json", "Accept": "application/json"},
        json={
            "query": """
                SELECT id, content, source, chunk_idx,
                       vector::similarity::cosine(embedding, $query) AS score
                FROM chunks
                WHERE embedding <|$k, 40|> $query
                ORDER BY score DESC
                LIMIT $k
            """,
            "vars": {
                "query": query_embedding.tolist(),
                "k":     k,
            }
        }
    )
    response.raise_for_status()
    results = response.json()[0]["result"]
    return results

chunks_retrieved = retrieve_chunks("How does IndentiaDB handle vector indexing?", k=5)
for c in chunks_retrieved:
    print(f"[{c['score']:.3f}] {c['content'][:100]}...")

Step 5: Assemble Context and Call LLM¶

import openai   # or any other LLM client

def answer_question(question: str, k: int = 5) -> str:
    """Retrieve relevant context and answer a question using an LLM."""
    relevant_chunks = retrieve_chunks(question, k=k)

    # Build context string
    context_parts = []
    for i, chunk in enumerate(relevant_chunks, 1):
        context_parts.append(
            f"[{i}] (relevance: {chunk['score']:.2f})\n"
            f"Source: {chunk['source']}, chunk {chunk['chunk_idx']}\n"
            f"{chunk['content']}"
        )
    context = "\n\n---\n\n".join(context_parts)

    prompt = f"""You are a helpful assistant. Answer the question using ONLY the provided context.
If the context does not contain enough information, say so explicitly.

Context:
{context}

Question: {question}

Answer:"""

    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,
        max_tokens=512,
    )

    return response.choices[0].message.content

answer = answer_question("What HNSW parameters should I use for a 10M document corpus?")
print(answer)

Knowledge Graph RAG¶

IndentiaDB uniquely combines vector similarity retrieval with RDF-star provenance annotations. This enables RAG with confidence-weighted context — the LLM receives not just relevant text but also the source and confidence of each knowledge graph fact.

Querying RDF-star Provenance for RAG Context¶

PREFIX ig:    <http://indentiadb.nl/ontology/>
PREFIX foaf:  <http://xmlns.com/foaf/0.1/>
PREFIX ex:    <http://example.org/>

SELECT ?subject ?predicate ?object ?source ?confidence ?lastVerified
WHERE {
    << ?subject ?predicate ?object >>
        ig:source       ?source ;
        ig:confidence   ?confidence ;
        ig:lastVerified ?lastVerified .

    FILTER (?confidence > 0.7)
}
ORDER BY DESC(?confidence)
LIMIT 10

Storing Facts with Provenance¶

PREFIX ig:  <http://indentiadb.nl/ontology/>
PREFIX ex:  <http://example.org/>

INSERT DATA {
    GRAPH <http://example.org/knowledge> {
        ex:IndentiaDB ex:hasFeature ex:HNSWIndex .

        << ex:IndentiaDB ex:hasFeature ex:HNSWIndex >>
            ig:source       <https://arxiv.org/abs/1603.09320> ;
            ig:confidence   0.95 ;
            ig:lastVerified "2025-09-01T00:00:00Z"^^xsd:dateTime ;
            ig:extractedBy  ex:AutoExtractorV2 .
    }
}

Complete Knowledge Graph RAG Pipeline¶

import requests

SPARQL_ENDPOINT = "http://localhost:7001/sparql"
CONFIDENCE_THRESHOLD = 0.7

def fetch_kg_context(topic_uri: str) -> list[dict]:
    """Fetch high-confidence RDF-star facts about a topic for RAG context."""
    sparql_query = f"""
        PREFIX ig:   <http://indentiadb.nl/ontology/>
        PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

        SELECT ?subject ?predName ?object ?source ?confidence WHERE {{
            << ?subject ?predicate ?object >>
                ig:source     ?source ;
                ig:confidence ?confidence .

            OPTIONAL {{ ?predicate rdfs:label ?predName . }}
            FILTER (?confidence > {CONFIDENCE_THRESHOLD})
            FILTER (STRSTARTS(STR(?subject), STR(<{topic_uri}>)))
        }}
        ORDER BY DESC(?confidence)
        LIMIT 20
    """

    response = requests.post(
        SPARQL_ENDPOINT,
        headers={
            "Content-Type": "application/sparql-query",
            "Accept":       "application/sparql-results+json",
        },
        data=sparql_query,
    )
    response.raise_for_status()
    bindings = response.json()["results"]["bindings"]

    facts = []
    for b in bindings:
        facts.append({
            "subject":    b["subject"]["value"],
            "predicate":  b.get("predName", b["predicate"])["value"],
            "object":     b["object"]["value"],
            "source":     b["source"]["value"],
            "confidence": float(b["confidence"]["value"]),
        })
    return facts


def kg_rag_answer(question: str, topic_uri: str) -> str:
    """Answer a question using KG facts + vector-retrieved chunks."""
    import openai

    # Fetch structured KG facts
    facts = fetch_kg_context(topic_uri)

    # Fetch unstructured relevant chunks
    query_embedding = model.encode([question], normalize_embeddings=True)[0]
    chunks = retrieve_chunks(question, k=3)

    # Build context
    fact_lines = "\n".join(
        f"  - [{f['confidence']:.0%} confidence] {f['subject']} {f['predicate']} "
        f"{f['object']} (source: {f['source']})"
        for f in facts
    )
    chunk_lines = "\n\n".join(c["content"] for c in chunks)

    context = f"""Structured Knowledge Graph Facts (confidence-weighted):
{fact_lines}

Unstructured Document Excerpts:
{chunk_lines}"""

    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": (
                f"Answer the question using the knowledge graph facts and document "
                f"excerpts below. Cite confidence levels where relevant.\n\n"
                f"{context}\n\nQuestion: {question}\nAnswer:"
            )
        }],
        temperature=0.1,
    )
    return response.choices[0].message.content


answer = kg_rag_answer(
    question="What is the HNSW index based on?",
    topic_uri="http://example.org/HNSWIndex"
)
print(answer)

Vector K-NN in LPG¶

In addition to the HNSW-backed vector search via SurrealQL and SPARQL, IndentiaDB supports K-nearest-neighbour queries directly on LPG node properties. This is useful when embeddings are stored as node properties in the LPG projection (e.g., derived from RDF literals or document fields).

How It Works¶

The VectorKnn query kind performs a brute-force scan over nodes that have the specified property, computes a similarity/distance score for each, and returns the top-k results above the (optional) threshold. No separate index is required — the scan runs in-process on the in-memory CSR graph.

For high-cardinality graphs, consider filtering with filter.label to reduce the candidate set before scoring.

Request¶

POST /lpg/query
Authorization: Bearer <token>
Content-Type: application/json

{
  "kind": {
    "VectorKnn": {
      "property": "embedding",
      "query_vector": [0.021, -0.039, 0.085, "..."],
      "k": 10,
      "metric": "cosine",
      "threshold": 0.75,
      "filter": {
        "label": "Document",
        "properties": { "language": "en" }
      }
    }
  },
  "return_fields": ["id", "title", "author"]
}

Response¶

{
  "rows": [
    { "id": "http://example.org/doc1", "title": "Graph Databases", "author": "Smith", "score": 0.94 },
    { "id": "http://example.org/doc2", "title": "Knowledge Graphs", "author": "Jones", "score": 0.88 }
  ],
  "total": 2
}

The score field is always included: - cosine / dot: higher is better - l2: lower is better (Euclidean distance)

Example: Semantic Document Search in LPG¶

# First, insert documents with embeddings via SPARQL
curl -X POST http://localhost:7001/update \
  -H "Content-Type: application/sparql-update" \
  -d '
  PREFIX ex: <http://example.org/>
  INSERT DATA {
    ex:doc1 a ex:Document ;
            ex:title     "Graph Databases" ;
            ex:embedding "[0.1, 0.2, 0.3]" .
    ex:doc2 a ex:Document ;
            ex:title     "Knowledge Graphs" ;
            ex:embedding "[0.15, 0.25, 0.28]" .
  }'

# Then query by vector similarity in the LPG
curl -X POST http://localhost:7001/lpg/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": {
      "VectorKnn": {
        "property": "embedding",
        "query_vector": [0.12, 0.22, 0.31],
        "k": 5,
        "metric": "cosine",
        "threshold": 0.8,
        "filter": { "label": "http://example.org/Document" }
      }
    },
    "return_fields": ["id", "title"]
  }'

Combining VectorKnn with Graph Algorithms¶

A powerful pattern is to retrieve semantically similar nodes with VectorKnn, then run a graph algorithm on the resulting subgraph. For example:

1. VectorKnn → find the 50 most similar Article nodes to a query embedding
2. Extract node IDs from the response
3. POST /algo/pagerank → rank those 50 nodes by their graph-structural importance

This combines semantic relevance (vector similarity) with structural authority (PageRank) for superior result ranking.