Skip to content
TensorDB

Configuration

TensorDB is configured through the Config struct. All parameters have sensible defaults — you can start with Config::default() and tune as needed.

Basic Usage

use tensordb::{Database, Config};


// Use defaults
let db = Database::open("data", Config::default())?;


// Custom configuration
let mut config = Config::default();
config.memtable_max_bytes = 8 * 1024 * 1024;  // 8MB memtable
config.shard_count = 8;                         // 8 shards
config.fast_write_enabled = true;               // Direct write path


let db = Database::open("data", config)?;

Configuration Reference

Search and browse all configuration parameters:

Storage
wal_fsync_every_n_records usize 128
Number of WAL records before issuing an fsync. Lower values increase durability but reduce write throughput.
Constraint: > 0
memtable_max_bytes usize 4MB
Maximum size of the in-memory memtable before flushing to an SSTable. Larger values reduce flush frequency but increase memory usage.
Constraint: >= 1024
sstable_block_bytes usize 16KB
Size of each data block within SSTables. Larger blocks improve sequential read throughput; smaller blocks reduce read amplification for point lookups.
Constraint: >= 512
sstable_max_file_bytes u64 64MB
Maximum size of a single SSTable file. Controls the granularity of compaction units.
bloom_bits_per_key usize 10
Number of bits allocated per key in bloom filters. 10 bits gives ~1% false positive rate. Higher values reduce false positives at the cost of memory.
Constraint: >= 4
Caching
block_cache_bytes usize 32MB
Total memory budget for the SSTable block cache (LRU). Caches decompressed data blocks for fast repeated access.
index_cache_entries usize 1024
Maximum number of SSTable index blocks to cache. Each entry maps to one SSTable's block index.
Compaction
compaction_l0_threshold usize 8
Number of L0 SSTables that trigger compaction into L1. Lower thresholds keep L0 small but increase compaction frequency.
Constraint: > 0
compaction_l1_target_bytes u64 10MB
Target total size for Level 1. Higher levels are sized as L1 × size_ratio^(level-1).
compaction_size_ratio u64 10
Size multiplier between adjacent levels. A ratio of 10 means L2 targets 100MB, L3 targets 1GB, etc.
compaction_max_levels usize 7
Maximum number of LSM levels (L0 through L6). More levels allow more data but increase worst-case read amplification.
Constraint: >= 2
Sharding
shard_count usize 4
Number of independent shards. Keys are routed by hash(key) % shard_count. More shards improve write parallelism but increase resource usage.
Constraint: > 0
Fast Write Path
fast_write_enabled bool true
Enable the direct fast write path that bypasses the shard actor channel. Provides ~20× write speedup. Falls back to channel path when memtable is full or subscribers are active.
fast_write_wal_batch_interval_us u64 1000
Interval in microseconds for the group WAL durability thread to batch and fsync writes. Lower values reduce latency; higher values improve throughput.
AI Runtime
ai_auto_insights bool false
Enable automatic insight synthesis from write patterns. The AI runtime batches events and generates insights stored under the __ai/ key prefix.
ai_batch_window_ms u64 20
Time window in milliseconds for batching AI events before processing. Only used when ai_auto_insights is enabled.
Constraint: > 0 when ai_auto_insights enabled
ai_batch_max_events usize 16
Maximum number of events per AI batch. When reached, the batch is processed immediately without waiting for the full window.
Constraint: > 0 when ai_auto_insights enabled
ai_inline_risk_assessment bool false
Enable inline risk scoring on every write. Adds a risk_score annotation to write results based on anomaly detection.
ai_annotate_reads bool false
Enable AI annotations on read results. Adds contextual metadata like access frequency and staleness indicators.
ai_compaction_advisor bool false
Enable the AI compaction advisor. Monitors compaction patterns and suggests optimal compaction timing and level targets.
ai_cache_advisor bool false
Enable the AI cache advisor. Analyzes access patterns and recommends cache sizing and eviction policy adjustments.
ai_access_stats_size usize 1024
Number of key access records to retain for AI analysis. Used by query advisor, cache advisor, and access pattern analysis.

Tuning Profiles

High-Throughput Writes

For write-heavy workloads (event logging, time-series ingestion):

let mut config = Config::default();
config.memtable_max_bytes = 16 * 1024 * 1024;  // 16MB — flush less often
config.wal_fsync_every_n_records = 512;          // Batch more WAL syncs
config.fast_write_enabled = true;                // Direct path (default)
config.fast_write_wal_batch_interval_us = 2000;  // 2ms batch window
config.shard_count = 8;                          // More parallelism

Low-Latency Reads

For read-heavy workloads (OLTP, key-value lookups):

let mut config = Config::default();
config.block_cache_bytes = 128 * 1024 * 1024;   // 128MB block cache
config.index_cache_entries = 4096;                // Cache more indexes
config.bloom_bits_per_key = 14;                   // Lower false positives
config.compaction_l0_threshold = 4;               // Compact sooner

Validation

TensorDB validates configuration at startup. Invalid configurations return a TensorError::Config error:

let mut config = Config::default();
config.memtable_max_bytes = 100;  // Too small! Must be >= 1024


match Database::open("data", config) {
    Err(tensordb::TensorError::Config(msg)) => {
        eprintln!("Invalid config: {}", msg);
    }
    _ => {}
}