Compact RDF storage and LZ4¶

In the RDF source-of-truth path, a document is ingested once into the canonical RDF graph, including its text. Document, full-text and vector indexes are derived projections. Changes and deletes follow the same canonical write path. This contract requires the RDF projection mode; an independent Elasticsearch store must not be discarded as though it were rebuildable from RDF.

What changed¶

The triples projection uses SHA-256 subject/object index keys instead of copying full literals into four index keys. Full values, datatypes, language information and RDF 1.2 triple terms remain in records. Lookups verify the original value as well as its hash. The redundant idx_spo index is removed.

Migration backfills 128 records at a time with a durable cursor, builds replacement indexes and drains legacy index data in bounded transactions before removing their definitions. Other application tables are preserved. The storage engine also validates manifest overlap by key range and synchronizes new compaction files, manifests and value-log rotation before publishing references to them.

Recovery must not retire a WAL segment while only part of it has reached an SST file. The maintenance build includes a regression for a second crash during recovery. Record-range scans also stream before applying an outer LIMIT: small migration pages no longer first collect the full remaining table in RAM.

The subsequent compaction fix (29dbe38c1) drains remaining level debt after each merge, including after reopening a store. It avoids waiting for another flush while full L0 files already block writes. Bottom-level rewrites run after that debt and do not loop indefinitely. A regression reproduces the blocked write; all 1,038 storage tests passed locally (three ignored). This fix is a separate release from the existing devtest maintenance image.

Compression is optional and enabled by default¶

# Set this in the server environment to disable compression for new writes.
SURREAL_SURREALKV_VLOG_COMPRESSION=false

Adaptive LZ4 applies to values of at least 256 bytes when it saves approximately 12.5% or more; otherwise the value is stored raw. Raw and compressed files can coexist. Checksums, decoded-length limits and format validation reject corrupt frames. Changing the setting starts a file with the selected codec.

Disabling compression does not rewrite existing data. The new binary still reads compressed files, but a previous binary without this format support does not. A downgrade requires a compatible backup or a verified uncompressed repack; RDF schema and Raft format compatibility must also be checked independently.

Offline migration and reclaim¶

Stop the process using the local store and take a recoverable backup first. Confirm the namespace/database and that historical MVCC/versioned storage was not enabled. Run this maintenance on one replica at a time while the remaining voters retain quorum. The paths below are examples; the destination must be a new sibling directory on the same filesystem.

bws run -- indentiagraph storage migrate-triples \
  --path /data/surrealdb --namespace indentiagraph --database alerting

bws run -- indentiagraph storage repack \
  --source /data/surrealdb \
  --destination /data/surrealdb-compacted \
  --source-is-non-versioned

repack defaults to LZ4. Add --lz4 false to write an uncompressed destination. It acquires the source lock and refuses an existing destination. It copies all live raw keys/values, not just triples, in transactions of at most 512 records or approximately 8 MiB. After reopening, it verifies every key/value byte through a length-delimited SHA-256 digest, entry count and logical byte count, then writes REPACK-VERIFIED.json. The source is retained and the CLI does not swap paths.

Only install the verified destination while the server is stopped. Keep the old store until restart, Raft catch-up, projection certification, queries and update/delete checks succeed. Then remove the verified obsolete copies and measure actual free space. Preserve the active Raft directory and any independent ES store. Reflink backups share blocks, so directory sizes alone do not prove physical space has been freed.

Fix 16d6e553a preserves unbounded transactional scan ranges. Older repack binaries could copy zero records and falsely verify the empty output because copying and verification shared the same iterator defect. A report with entries: 0 from those binaries is invalid: do not install it or delete backups. The corrected command refuses empty output; regressions check 600 known records after reopening and forward/reverse scans with pending transaction writes. If an empty output was already installed, restore the retained store while its server is stopped, preserve both directories, then validate data and projections before continuing maintenance on another replica. This repair is not proof of completed migration or reclaimed space on a running cluster.

Measured scope¶

These are separate experiments, not additive savings or production forecasts. Measure build scratch space, peak memory, sustained ingestion and recovery on the actual storage before choosing shard capacity.