Architecture

TensorDB is a single-node, append-only, bitemporal ledger database. Every component is designed around one principle: data is immutable.

TensorDB Architecture

📱

Client

SQL / Rust API / Python / Node.js

▼

⚡

Database Engine

SQL Parser → Planner → Executor

▼ hash(key) % shard_count

Shard 0

Single-writer actor

Shard 1

Single-writer actor

Shard 2

Single-writer actor

Shard 3

Single-writer actor

▼

Write Path

WAL

CRC-framed, fsync

→

Memtable

Skip list, 4MB

→

Flush

SSTable L0

▼

LSM Storage

L0
L1
L2
L3

← Compaction merges levels →

Read Path

Cache

Block + Index LRU

→

Bloom Filter

10 bits/key

→

Temporal Filter

AS OF / VALID AT

AI Runtime (Tier-0)

Risk Scoring

Inline, per-write

Insights

Batched synthesis

Advisors

Query, Cache, Compaction

Core Layers

Database Engine

The Database struct is the public API entry point. It handles:

SQL parsing and execution — Parses SQL statements, generates query plans, and executes them
Shard routing — Routes writes to the correct shard via hash(key) % shard_count
Configuration — Manages all tuning knobs through the Config struct

Shards

Each shard is a single-writer actor that owns:

A WAL (write-ahead log) handle
An in-memory memtable (skip list)
A local commit counter (monotonically increasing)

Shards are independent — no coordination between shards is needed for writes. This enables high write throughput through parallelism.

Storage Engine (LSM-Tree)

Data flows through the LSM tree:

WAL — Every write is first appended to the write-ahead log with CRC framing
Memtable — The write is then inserted into the in-memory skip list
Flush — When the memtable reaches memtable_max_bytes (default 4MB), it’s flushed to an L0 SSTable
Compaction — L0 SSTables are merged into L1, L1 into L2, etc., up to L6

Design Principles

Principle	Implementation
Immutability	No in-place updates; all data is append-only
Temporal completeness	Every fact has system time + business time
Crash safety	WAL fsync before acknowledging writes
Lock-free reads	`ShardReadHandle` bypasses the actor channel
Zero-copy where possible	`mmap` reads for SSTable access