SEDB — Smart & Expandable Database

A feature-rich vector/embeddings database built in Kotlin with hybrid search, RAG support, multi-tenant authentication, and dual HTTP/gRPC APIs.

Features

• Multiple Index Types — Flat (brute-force), HNSW (approximate), Product Quantization (memory-efficient), Scalar Quantization (INT8/INT4), HNSW+SQ (graph-based search with scalar quantization), Memory-mapped ( larger-than-RAM), IVF (inverted file with k-means clustering), Binary Quantization (1-bit per dimension, Hamming distance), IVF-PQ (IVF partitioning + PQ compression), DiskANN (SSD-friendly Vamana graph with PQ navigation for billion-scale datasets), HNSW-IVF (HNSW-accelerated coarse quantizer for IVF partitioning)
• Dual API — HTTP REST (Ktor) and gRPC with full feature parity
• Full-Text Search — Inverted index with BM25 scoring, phrase search, wildcard search, faceted search, and autocomplete
• Graph Store — Relationship layer with edge CRUD, traversal, visualization, and filtering by label/direction
• RAG Pipeline — Document chunking, hybrid search (vector + BM25), re-ranking
• Collections & Namespaces — Isolated collections with per-collection index config, schema validation, schema migration, namespace partitioning, and collection aliases for zero-downtime reindexing
• Dynamic Index Tuning — Change HNSW efSearch at runtime to trade off recall vs. latency without rebuilding
• Multi-Tenancy — API key authentication with role-based access control (ADMIN, READWRITE, READONLY), collection-level RBAC, and tenant quotas
• Persistence — SQLite and PostgreSQL backends with Write-Ahead Log (WAL) for crash recovery
• Snapshots & Backups — Point-in-time snapshots, scheduled backups, point-in-time recovery, and S3 offloading
• Clustering — Leader election, replication, and sharding for high availability
• Observability — Prometheus metrics, query logging, slow query log, distributed tracing (OpenTelemetry/Jaeger), per-collection metrics, admin dashboard
• GUI Frontend — Built-in web UI at /gui with tabs for records, search, collections, ingest, graph, snapshots, and tenants
• Document Ingestion — Plain text, Markdown, HTML, CSV, JSON, JSON Lines, and images (via vision models)
• Multiple Embedding Providers — FastEmbed local ONNX (default), Ollama, OpenAI, Cohere, and HuggingFace
• CLIP Multi-modal Search — Native image embedding via CLIP for image-to-image and text-to-image search
• Temporal Search — Time-windowed vector search, point-in-time lookups, vector drift detection, and time-bucket aggregation
• Webhooks — Event-driven notifications with configurable retries and delivery logging
• Streaming — Server-Sent Events (SSE) for real-time search, ingestion, and change data capture (CDC)
• Encryption — Optional field-level AES encryption for record content and embeddings, with key rotation support
• Security Hardening — API key rotation with grace periods, HMAC-SHA256 webhook signing, encryption key versioning
• Named Vectors — Store multiple named vector fields per record (e.g., title_vector, body_vector) with per-vector search support
• Payload Indexing — Secondary indexes on metadata fields (keyword, numeric, text) for fast filtered search without full scans
• Geo-Spatial Search — Grid-based spatial index with radius, bounding-box, and polygon queries using the Haversine formula
• Scroll / Iterator API — Cursor-based pagination for iterating over large result sets without offset degradation
• Record Count — Lightweight count endpoints for both global and per-collection record counts
• Audit Logging — Immutable audit trail of administrative actions
• Rate Limiting — Configurable per-tier rate limits (public, standard, write, generation, admin)
• Graceful Shutdown — Drain mode with configurable grace period and timeout
• Kubernetes Operator — SedbCluster CRD with automated scaling (HPA), cron-based backup scheduling, and zero-downtime rolling upgrades with automatic rollback
• Data Import/Export — Bulk import/export in Parquet, CSV, and JSONL formats for migration between systems

Quick Start

Prerequisites

• JDK 21 (or later)
• No external services required — FastEmbed (local ONNX) is the default embedding provider. Optionally configure Ollama, OpenAI, Cohere, or HuggingFace.

Run with Gradle

./gradlew run

The server starts with:

• HTTP API on http://localhost:8080/api/v1
• gRPC on localhost:50051
• Admin dashboard at http://localhost:8080/admin
• GUI frontend at http://localhost:8080/gui
• API docs at http://localhost:8080/

Run with Docker

docker compose up

Deploy with Helm (Kubernetes)

helm install sedb ./helm/sedb

See helm/sedb/README.md for the full list of configurable values and examples.

Deploy with the Kubernetes Operator

# Install the SEDB operator
helm install sedb-operator ./helm/sedb-operator

# Create a managed SEDB cluster
kubectl apply -f ./helm/sedb-operator/examples/basic-cluster.yaml

The operator manages SedbCluster custom resources and provides automated scaling (HPA), cron-based backup scheduling (with S3 offloading), and zero-downtime rolling upgrades with automatic rollback. See helm/sedb-operator/README.md for the full CRD reference and examples.

Build a fat JAR

./gradlew shadowJar
java --add-modules jdk.incubator.vector -jar build/libs/sedb.jar

Configuration

All options can be set via CLI arguments or environment variables. CLI arguments take precedence. Run java -jar build/libs/sedb.jar --help for the full list.

Core Options

Option	Env Variable	Default	Description
--http-port	SEDB_HTTP_PORT	8080	HTTP API port
--grpc-port	SEDB_GRPC_PORT	50051	gRPC port
--host	SEDB_HOST	0.0.0.0	Bind host
--ollama-url	SEDB_OLLAMA_URL	http://localhost:11434	Ollama base URL
--embedding-model	SEDB_EMBEDDING_MODEL	paraphrase-multilingual:278m-mpnet-base-v2-fp16	Embedding model
--index-type	SEDB_INDEX_TYPE	flat	Index type: flat, hnsw, pq, sq_int8, sq_int4, hnsw_sq, hnsw_sq_int4, mmap, ivf, bq, ivf_pq, diskann, hnsw_ivf
--distance-metric	SEDB_DISTANCE_METRIC	cosine	Distance metric: cosine, euclidean, inner_product, manhattan, hamming, chebyshev
--auth	SEDB_AUTH	true	Enable API key authentication
--wal	SEDB_WAL	true	Enable Write-Ahead Log
--cache	SEDB_CACHE	true	Enable query cache

Persistence

Option	Env Variable	Default	Description
--data-dir	SEDB_DATA_DIR	data	Data directory
--sqlite	SEDB_SQLITE	default	Use SQLite persistence
--sqlite-path	SEDB_SQLITE_PATH	data/sedb.db	SQLite database path
--postgres	SEDB_POSTGRES	—	Use PostgreSQL persistence
--postgres-url	SEDB_POSTGRES_URL	jdbc:postgresql://localhost:5432/sedb	PostgreSQL JDBC URL
--snapshots	—	false	Enable point-in-time snapshots

Security

Option	Env Variable	Default	Description
--tls / --no-tls	SEDB_TLS	false	Enable TLS for HTTP and gRPC
--tls-cert	SEDB_TLS_CERT	—	Path to PEM certificate file
--tls-key	SEDB_TLS_KEY	—	Path to PEM private key file (PKCS8)
--encryption	SEDB_ENCRYPTION_ENABLED	false	Enable field-level encryption
--encryption-key	SEDB_ENCRYPTION_KEY	—	Encryption passphrase

Embedding Providers

Option	Env Variable	Default	Description
--embedding-provider	SEDB_EMBEDDING_PROVIDER	fastembed	Provider: fastembed, ollama, openai, cohere, huggingface
--openai-api-key	SEDB_OPENAI_API_KEY	—	OpenAI API key
--openai-model	SEDB_OPENAI_MODEL	text-embedding-3-small	OpenAI model
--cohere-api-key	SEDB_COHERE_API_KEY	—	Cohere API key
--cohere-model	SEDB_COHERE_MODEL	embed-english-v3.0	Cohere model
--huggingface-api-key	SEDB_HUGGINGFACE_API_KEY	—	HuggingFace API token
--huggingface-model	SEDB_HUGGINGFACE_MODEL	sentence-transformers/all-MiniLM-L6-v2	HuggingFace model
--fastembed-model	SEDB_FASTEMBED_MODEL	nomic-ai/nomic-embed-text-v1.5	FastEmbed ONNX model name
--fastembed-cache-dir	SEDB_FASTEMBED_CACHE_DIR	~/.cache/sedb-fastembed/	FastEmbed model cache directory
--fastembed-max-length	SEDB_FASTEMBED_MAX_LENGTH	512	FastEmbed max token length
--fastembed-threads	SEDB_FASTEMBED_THREADS	auto	FastEmbed ONNX inference threads

CLIP Multi-modal Embedding

Enable CLIP to embed images directly into vectors for cross-modal search (image-to-image, text-to-image).

Option	Env Variable	Default	Description
--clip-enabled	SEDB_CLIP_ENABLED	false	Enable CLIP multi-modal embedding
--clip-api-key	SEDB_CLIP_API_KEY	—	API key for CLIP embedding service
--clip-model	SEDB_CLIP_MODEL	clip-vit-base-patch32	CLIP model name
--clip-base-url	SEDB_CLIP_BASE_URL	https://api.openai.com/v1	CLIP API base URL

Backup & S3

Option	Env Variable	Default	Description
--backup	SEDB_BACKUP	false	Enable automatic backups
--backup-interval-ms	SEDB_BACKUP_INTERVAL_MS	3600000	Backup interval in milliseconds
--s3	SEDB_S3	false	Enable S3 backup
--s3-endpoint	SEDB_S3_ENDPOINT	https://s3.amazonaws.com	S3 endpoint URL
--s3-bucket	SEDB_S3_BUCKET	sedb-backups	S3 bucket name

Cluster (HA)

Option	Env Variable	Default	Description
--cluster	SEDB_CLUSTER	false	Enable cluster mode
--cluster-node-id	SEDB_CLUSTER_NODE_ID	node-1	Unique node identifier
--cluster-seeds	SEDB_CLUSTER_SEEDS	—	Comma-separated seed node addresses

Observability

Option	Env Variable	Default	Description
--audit	SEDB_AUDIT	false	Enable audit logging
--tracing	SEDB_TRACING	false	Enable distributed tracing
--tracing-exporter-type	SEDB_TRACING_EXPORTER_TYPE	otlp	Exporter: otlp, jaeger, zipkin
--slow-query-threshold-ms	SEDB_SLOW_QUERY_THRESHOLD_MS	100	Slow query threshold in ms
--log-format	SEDB_LOG_FORMAT	text	Log output format: text or json

Structured Logging

SEDB supports JSON-formatted log output for production log aggregation systems (ELK, Loki, etc.).

Set SEDB_LOG_FORMAT=json (or --log-format json) to switch from plain-text to structured JSON logs. Every HTTP request is automatically assigned a correlation ID (UUID), available as:

• correlationId field in JSON log output and MDC
• X-Request-Id response header (echoed back to the caller)

If the incoming request includes an X-Request-Id header, that value is used instead of generating a new one.

JSON output example:

{"timestamp":"2025-01-15T10:30:00.123Z","level":"INFO","logger":"t.l.sedb.server.HttpServer","message":"GET /api/v1/search 200","thread":"ktor-netty-worker-1","correlationId":"abc-123"}

Text output (default) with correlation ID:

10:30:00.123 [ktor-netty-worker-1] INFO  t.l.sedb.server.HttpServer - [abc-123] GET /api/v1/search 200

Record TTL (Time-to-Live)

Records can be given a TTL so they automatically expire after a configurable duration. Pass ttlSeconds when adding or upserting a record (via REST, gRPC, or the client SDK). Expired records are immediately filtered out of get, all, search, and scroll results.

A background TTL reaper runs periodically to physically remove expired records and reclaim space. Compaction also removes expired records.

Option	Env Variable	Default	Description
--ttl-reaper	SEDB_TTL_REAPER	true	Enable/disable the background TTL reaper
--ttl-reaper-interval-ms	SEDB_TTL_REAPER_INTERVAL_MS	60000	Reaper interval in milliseconds

API Overview

Records

Method	Endpoint	Description
GET	/api/v1/records?limit=100&offset=0	List records (paginated)
GET	/api/v1/records/{id}	Get a record by ID
POST	/api/v1/records	Add a record
POST	/api/v1/records/batch	Add records in batch
PUT	/api/v1/records	Upsert (update or insert)
PATCH	/api/v1/records/{id}	Partial update a record
DELETE	/api/v1/records/{id}	Delete a record
DELETE	/api/v1/records	Clear all records
POST	/api/v1/records/bulk-delete	Bulk delete by IDs
GET	/api/v1/records/count	Get total record count
POST	/api/v1/records/scroll	Scroll through records

Search

Method	Endpoint	Description
POST	/api/v1/search	Semantic search
POST	/api/v1/search/batch	Batch search
POST	/api/v1/search/multi-vector	Multi-vector search
POST	/api/v1/search/hybrid	Hybrid search (vector + BM25)
POST	/api/v1/search/sparse	Sparse vector search
POST	/api/v1/search/sparse/index	Index a sparse vector record
POST	/api/v1/search/image	CLIP image search (image-to-image / text-to-image)
POST	/api/v1/records/clip	Ingest an image via CLIP embedding

Temporal Search

Method	Path	Description
POST	/api/v1/temporal/search	Time-windowed vector search
GET	/api/v1/temporal/records/{id}/at	Point-in-time record lookup
GET	/api/v1/temporal/records/{id}/drift	Vector drift between two timestamps
POST	/api/v1/temporal/aggregate	Time-bucket aggregation of search results

Collections

Method	Endpoint	Description
POST	/api/v1/collections	Create a collection
GET	/api/v1/collections	List collections
GET	/api/v1/collections/{name}	Get collection stats
DELETE	/api/v1/collections/{name}	Drop a collection
POST	/api/v1/collections/{name}/records	Add a record to a collection
PUT	/api/v1/collections/{name}/records	Upsert a collection record
PATCH	/api/v1/collections/{name}/records/{id}	Partial update a collection record
PUT	/api/v1/collections/{name}/records/batch	Batch upsert collection records
GET	/api/v1/collections/{name}/records	List collection records
POST	/api/v1/collections/{name}/search	Search within a collection
GET	/api/v1/collections/{name}/schema	Get collection schema
POST	/api/v1/collections/{name}/schema/validate	Validate a record against schema
POST	/api/v1/collections/{name}/schema/migrate	Migrate collection schema
GET	/api/v1/collections/{name}/namespaces	List namespaces
GET	/api/v1/collections/{name}/namespaces/{ns}/records	List namespace records
POST	/api/v1/collections/{name}/namespaces/{ns}/records	Add a record to a namespace
POST	/api/v1/collections/{name}/namespaces/{ns}/search	Search within a namespace
DELETE	/api/v1/collections/{name}/namespaces/{ns}	Clear a namespace
GET	/api/v1/collections/{name}/count	Get collection record count
POST	/api/v1/collections/{name}/scroll	Scroll through collection records
POST	/api/v1/collections/{name}/search/named-vector	Search by named vector field
POST	/api/v1/collections/{name}/payload-indexes	Create a payload index
GET	/api/v1/collections/{name}/payload-indexes	List payload indexes
DELETE	/api/v1/collections/{name}/payload-indexes/{field}	Drop a payload index
POST	/api/v1/collections/{name}/geo-indexes	Create a geo index
GET	/api/v1/collections/{name}/geo-indexes	List geo indexes
DELETE	/api/v1/collections/{name}/geo-indexes/{field}	Drop a geo index
POST	/api/v1/collections/{name}/geo-search/radius	Geo radius search
POST	/api/v1/collections/{name}/geo-search/bounding-box	Geo bounding box search
POST	/api/v1/collections/{name}/geo-search/polygon	Geo polygon search
PATCH	/api/v1/collections/{name}/index/params	Update tunable index parameters
PATCH	/api/v1/index/params	Update default DB index parameters

Cross-Collection Search

Method	Endpoint	Description
POST	/api/v1/collections/search	Search across multiple collections

Search across multiple (or all) collections in a single query. Results are merged by score and annotated with the source collection name.

POST /api/v1/collections/search
{
  "queryEmbedding": [0.1, 0.2, 0.3],
  "collections": ["col1", "col2"],
  "topK": 10,
  "filter": {"key": "value"},
  "scoreThreshold": 0.5
}

• collections — optional; omit or pass empty to search all collections.
• scoreThreshold — optional minimum similarity score.
• Non-existent collection names are silently skipped.

Data Import/Export

Method	Endpoint	Description
GET	/api/v1/export?format={jsonl|csv|parquet}	Export all default DB records
POST	/api/v1/import?format={jsonl|csv|parquet}	Import records into default DB
GET	/api/v1/collections/{name}/export?format={jsonl|csv|parquet}	Export all collection records
POST	/api/v1/collections/{name}/import?format={jsonl|csv|parquet}	Import records into a collection

Supported formats: jsonl, csv, parquet. Export returns a file download; import accepts the file as the request body and returns {"imported": <count>}.

Collection Aliases

Method	Endpoint	Description
POST	/api/v1/aliases	Create or update a collection alias
GET	/api/v1/aliases	List all aliases
GET	/api/v1/aliases/{alias}	Get alias details
DELETE	/api/v1/aliases/{alias}	Delete an alias

Query DSL & Aggregations

Method	Endpoint	Description
POST	/api/v1/query	Query records with filters
POST	/api/v1/aggregate	Aggregate over records

Versioned Records

Method	Endpoint	Description
GET	/api/v1/records/{id}/versions	List record version history
GET	/api/v1/records/{id}/versions/{version}	Get a specific record version
POST	/api/v1/records/{id}/rollback	Rollback to a previous version

Partial Updates (PATCH)

The PATCH /api/v1/records/{id} endpoint allows partial updates to an existing record. All request fields are optional — only the provided fields are updated:

• metadata — replaces metadata without re-embedding.
• content — replaces content and triggers re-embedding.
• model — embedding model override (only used when content is provided).
• ttlSeconds — updates the TTL.
• encrypted — updates the encryption flag.

If the record is not found, a 404 is returned. A collection variant is available at PATCH /api/v1/collections/{name}/records/{id} with an optional embedding field instead of model.

Conditional Updates (Optimistic Concurrency)

The PUT /api/v1/records endpoint supports conditional updates via an optional ifVersion field. When provided, the update only succeeds if the record's current version matches ifVersion (compare-and-swap semantics).

PUT /api/v1/records
{
  "id": "rec-1",
  "content": "updated content",
  "ifVersion": 2
}

• Success → 200 OK with the updated record and new version in the response.
• Version mismatch → 409 Conflict with the current version in the error response.
• Record not found → 409 Conflict (no version to match).
• Versioned store disabled → 400 Bad Request.
• No ifVersion → unconditional upsert (backward compatible).

The gRPC AddRecord RPC also supports if_version (optional field) with ABORTED status on conflict.

Full-Text Search

Method	Endpoint	Description
POST	/api/v1/fts/index	Index a document
POST	/api/v1/fts/index/batch	Batch index documents
DELETE	/api/v1/fts/index/{id}	Remove a document
POST	/api/v1/fts/search	Search documents
POST	/api/v1/fts/search/phrase	Phrase search
POST	/api/v1/fts/search/wildcard	Wildcard search
POST	/api/v1/fts/suggest	Autocomplete suggestions
GET	/api/v1/fts/stats	FTS index statistics

Graph

Method	Endpoint	Description
POST	/api/v1/graph/edges	Add an edge
GET	/api/v1/graph/edges	List all edges
GET	/api/v1/graph/edges/{id}	Get an edge by ID
DELETE	/api/v1/graph/edges/{id}	Delete an edge
GET	/api/v1/graph/nodes/{nodeId}/edges	Get edges for a node
POST	/api/v1/graph/traverse	Traverse the graph
GET	/api/v1/graph/stats	Graph statistics
GET	/api/v1/graph/visualization	Graph visualization data

Ingestion

Method	Endpoint	Description
POST	/api/v1/ingest/document	Ingest a document (text, MD, HTML, CSV, JSON, JSONL)
POST	/api/v1/ingest/image	Ingest an image via vision model
POST	/api/v1/ingest/stream	Stream-ingest records (NDJSON response)
GET	/api/v1/ingest/formats	List supported ingestion formats

Generation

Method	Endpoint	Description
POST	/api/v1/generate	Text generation
POST	/api/v1/summarize	Text summarization
POST	/api/v1/describe-image	Image description

Snapshots

Method	Endpoint	Description
GET	/api/v1/snapshots	List snapshots
POST	/api/v1/snapshots	Create a snapshot
POST	/api/v1/snapshots/{id}/restore	Restore a snapshot
DELETE	/api/v1/snapshots/{id}	Delete a snapshot

Webhooks

Method	Endpoint	Description
POST	/api/v1/webhooks	Register a webhook
GET	/api/v1/webhooks	List webhooks
GET	/api/v1/webhooks/{id}	Get a webhook
DELETE	/api/v1/webhooks/{id}	Delete a webhook
GET	/api/v1/webhooks/{id}/deliveries	Get delivery log

Streaming (SSE)

Method	Endpoint	Description
GET	/api/v1/stream/search	Stream search results via SSE
GET	/api/v1/stream/ingest	Stream ingestion progress via SSE
GET	/api/v1/stream/changes	Subscribe to change events via SSE
GET	/api/v1/cdc/stream	CDC event stream (NDJSON/SSE)

Health & Info

Method	Endpoint	Description
GET	/api/v1/health	Health check
GET	/api/v1/livez	Liveness probe
GET	/api/v1/readyz	Readiness check (Ollama connectivity, index status)
GET	/api/v1/startupz	Startup probe
GET	/api/v1/auth/whoami	Current API key identity
GET	/api/v1/models	List available embedding models
GET	/metrics	Prometheus metrics

Admin

Method	Endpoint	Description
POST	/api/v1/admin/tenants	Create a tenant
GET	/api/v1/admin/tenants	List tenants
GET	/api/v1/admin/tenants/{id}	Get a tenant
PUT	/api/v1/admin/tenants/{id}	Update a tenant
DELETE	/api/v1/admin/tenants/{id}	Delete a tenant
POST	/api/v1/admin/keys	Create an API key
GET	/api/v1/admin/keys/{tenantId}	List API keys for a tenant
DELETE	/api/v1/admin/keys/{key}	Revoke an API key
POST	/api/v1/admin/keys/rotate	Rotate an API key
POST	/api/v1/admin/permissions	Grant collection permission
GET	/api/v1/admin/permissions/{apiKey}	List collection permissions
DELETE	/api/v1/admin/permissions/{apiKey}/{col}	Revoke collection permission
GET	/api/v1/admin/dashboard	Dashboard summary
POST	/api/v1/admin/collections/{name}/reindex	Rebuild collection index
POST	/api/v1/admin/compact	Trigger index compaction
GET	/api/v1/admin/compact/status	Compaction status
PUT	/api/v1/admin/quotas/{tenantId}	Set tenant quota
GET	/api/v1/admin/quotas/{tenantId}	Get tenant quota
GET	/api/v1/admin/quotas/{tenantId}/usage	Get tenant usage
DELETE	/api/v1/admin/quotas/{tenantId}	Remove tenant quota
GET	/api/v1/admin/quotas	List all quotas
GET	/api/v1/admin/audit	Query audit log
POST	/api/v1/admin/s3/upload	Upload snapshot to S3
POST	/api/v1/admin/s3/download	Download snapshot from S3
GET	/api/v1/admin/s3/list	List S3 snapshots
POST	/api/v1/admin/drain	Initiate graceful drain
POST	/api/v1/admin/pitr/recover	Point-in-time recovery
GET	/api/v1/admin/pitr/recovery-points	List recovery points
GET	/api/v1/admin/traces	List recent traces
GET	/api/v1/admin/traces/{traceId}	Get a trace
GET	/api/v1/admin/traces/summary	Tracing summary
GET	/api/v1/admin/slow-queries	Slow query log
GET	/api/v1/admin/index-builds	List index builds
GET	/api/v1/admin/index-builds/{buildId}	Get index build progress
GET	/api/v1/admin/metrics/collections	All collection metrics
GET	/api/v1/admin/metrics/collections/{name}	Per-collection metrics
POST	/api/v1/admin/backup/trigger	Trigger an immediate backup
GET	/api/v1/admin/backup/status	Get backup scheduler status
PUT	/api/v1/admin/backup/schedule	Update backup schedule
GET	/api/v1/admin/rate-limits	Get rate limit configuration
GET	/api/v1/admin/cluster/status	Get cluster status
POST	/api/v1/admin/encryption/rotate	Rotate encryption key
GET	/api/v1/admin/encryption/status	Encryption key status
POST	/api/v1/admin/encryption/reencrypt	Trigger re-encryption
GET	/api/v1/admin/cluster/node	Get current node info

Full OpenAPI documentation is available at src/main/resources/openapi/documentation.yaml.

Architecture

Diagram source (PlantUML)

@startuml
skinparam backgroundColor transparent
skinparam componentStyle rectangle
skinparam defaultFontName monospace

package "SEDB Server" {
  [HTTP API] as http
  [gRPC API] as grpc
  [Admin Dashboard] as admin

  package "Route Layer" {
    [Auth & Rate Limiting] as routes
  }

  package "Core" {
    [FTS Engine] as fts
    [SmartExpandable Database] as sedb
    [Graph Store] as graph
  }

  package "Vector Index" {
    [Flat | HNSW | PQ | Mmap] as idx
  }

  package "Storage" {
    [WAL] as wal
    [Persistence (SQLite / PostgreSQL)] as persist
    [Snapshots & Backups] as snap
  }

  [Ollama (Embeddings, Generation)] as ollama
}

http --> routes
grpc --> routes
admin --> routes
routes --> fts
routes --> sedb
routes --> graph
sedb --> idx
idx --> wal
idx --> persist
idx --> snap
sedb --> ollama
@enduml

Client Library

A Kotlin client library is available in the sedb-client/ module, supporting both HTTP and gRPC transports:

val client = SedbClient("http://localhost:8080")

// Add a record
val record = client.addRecord("Hello, world!", metadata = mapOf("source" to "example"))

// Semantic search
val results = client.search("greeting", topK = 5)

// Full-text search
val ftsResults = client.ftsSearch("hello", topK = 10, highlight = true)

// Collections
client.createCollection("my-collection", CollectionConfig(indexType = "hnsw"))
val stats = client.getCollectionStats("my-collection")

// Graph
val edge = client.addEdge(AddEdgeInput(sourceId = "a", targetId = "b", label = "related"))
val traversal = client.traverse(TraverseInput(startNodeId = "a", maxDepth = 3))

// Webhooks
client.registerWebhook(RegisterWebhookInput(url = "https://example.com/hook", events = listOf("RECORD_ADDED")))

// Snapshots
val snapshot = client.createSnapshot("backup-1", description = "Daily backup")
client.restoreSnapshot("backup-1")

// Admin
val tenant = client.createTenant("my-tenant")
val apiKey = client.createApiKey(tenant.id, role = "READWRITE")

The client supports hybrid search, sparse vector search, query DSL, aggregations, and versioned records:

// Hybrid search (vector + BM25)
val hybridResults = client.hybridSearch(HybridSearchInput(
    query = "greeting", topK = 5, vectorWeight = 0.7f, bm25Weight = 0.3f
))

// Sparse vector search
val sparseResults = client.sparseSearch(SparseSearchInput(
    sparseVector = SparseVectorInput(indices = listOf(0, 5, 12), values = listOf(0.5f, 1.2f, 0.8f)),
    topK = 10
))

// Query DSL with filters
val queryResult = client.queryRecords(query {
    eq("category", "science")
    numericGt("score", 0.5)
    sortBy("date", ascending = false)
    limit(20)
})

// Aggregations
val aggResult = client.aggregate(AggregateInput(
    groupByFields = listOf("category"), statsFields = listOf("author"), topN = 5
))

// Versioned records
val versions = client.getRecordVersions("record-id")
val oldVersion = client.getRecordVersion("record-id", version = 1)
val rollback = client.rollbackRecord("record-id", targetVersion = 1)

// Point-in-time recovery (admin)
val recoveryPoints = client.availableRecoveryPoints()
val snapshot = client.recoverToPointInTime("2026-03-28T12:00:00Z")

The client exposes connection pool configuration for both transports:

val client = SedbClient {
    host = "localhost"
    httpPort = 8080
    transport = Transport.HTTP

    // HTTP connection pool
    maxConnectionsTotal = 1000       // max simultaneous connections
    maxConnectionsPerRoute = 100     // max connections per host
    pipelining = false               // HTTP pipelining
    keepAliveTimeMs = 5_000          // keep-alive duration
    connectRetryAttempts = 5         // retry attempts on connect failure
}

val grpc = SedbClient {
    host = "localhost"
    grpcPort = 50051
    transport = Transport.GRPC

    // gRPC channel tuning
    maxInboundMessageSize = 8 * 1024 * 1024  // 8 MB
    keepAliveTimeMs = 5_000
}

The client includes built-in retry with exponential backoff and a circuit breaker for resilient communication:

val client = SedbClient {
    host = "localhost"
    httpPort = 8080

    // Retry with exponential backoff
    retry {
        maxAttempts = 5              // total attempts (default: 3)
        initialDelayMs = 200         // first retry delay (default: 100)
        maxDelayMs = 15_000          // delay cap (default: 10_000)
        multiplier = 2.0             // backoff multiplier (default: 2.0)
        jitterFactor = 0.1           // random jitter 0.0–1.0 (default: 0.1)
    }

    // Circuit breaker
    circuitBreaker {
        failureThreshold = 5         // failures before opening (default: 5)
        resetTimeoutMs = 30_000      // time before half-open probe (default: 30_000)
        halfOpenMaxAttempts = 1      // probes in half-open state (default: 1)
    }
}

By default, retries are triggered for HTTP 502/503/504/429 and IOException. Client errors (400/401/403/404) are never retried. Streaming methods (streamSearch, streamIngest, subscribeCdc) pass through without retry. You can supply a custom retryOn predicate for full control.

The client also supports SSE streaming over HTTP and gRPC streaming for real-time ingestion and CDC:

val grpcClient = SedbGrpcClient("localhost", 50051)

// Stream ingest
val results = grpcClient.streamIngest(flowOf(IngestItem("doc content")))
results.collect { println("Ingested: ${it.id}") }

// Subscribe to changes
grpcClient.subscribeCdc().collect { event ->
    println("${event.type}: ${event.recordId}")
}

FastEmbed Library (sedb-fastembed)

A standalone Kotlin/JVM library for local embedding generation using ONNX Runtime — no external API calls needed. Reimplements the core functionality of Python's fastembed.

Supported Modalities

• Dense text embeddings — TextEmbedding (e.g. nomic-ai/nomic-embed-text-v1.5, multilingual models)
• Sparse text embeddings — SparseTextEmbedding (SPLADE models)
• Late interaction — LateInteractionTextEmbedding (ColBERT models)
• Image embeddings — ImageEmbedding (CLIP, ResNet models)
• Cross-encoder reranking — TextCrossEncoder (MS MARCO models)

Standalone Usage

// Dense text embedding
val embedding = TextEmbedding(modelName = "nomic-ai/nomic-embed-text-v1.5")
val vector = embedding.embed("Hello, world!")          // FloatArray(768)
val batch = embedding.embed(listOf("doc1", "doc2"))    // List<FloatArray>
val query = embedding.queryEmbed("search query")       // adds model-specific prefix
embedding.close()

// Sparse embedding (SPLADE)
val sparse = SparseTextEmbedding()
val sparseVec = sparse.embed("Hello, world!")          // SparseEmbedding(indices, values)
sparse.close()

// Image embedding
val image = ImageEmbedding()
val imgVec = image.embedPath("/path/to/image.jpg")     // FloatArray(512)
image.close()

// Cross-encoder reranking
val reranker = TextCrossEncoder()
val scores = reranker.rerank("query", listOf("doc1", "doc2"))  // List<Float>
val ranked = reranker.rerankWithIndices("query", listOf("doc1", "doc2"))
reranker.close()

Models are auto-downloaded from HuggingFace Hub on first use and cached locally in ~/.cache/sedb-fastembed/. You can also point to a local directory containing pre-downloaded model files.

SEDB Server Integration

Set --embedding-provider fastembed to use local ONNX models instead of Ollama or cloud APIs:

java -jar sedb.jar --embedding-provider fastembed --fastembed-model nomic-ai/nomic-embed-text-v1.5

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

See LICENSE for the full text.

Copyright © 2026 LeNooby09