@d31ma/fylo

Filesystem-first document storage for Bun.

FYLO stores every document version as its own JSON file under a predictable filesystem path. Prefix indexes accelerate queries without duplicating data. The filesystem is the source of truth.

One Bun-native storage engine with schema versioning, field-level encryption, file-based RLS authorization, WORM append-only mode, local queues, and sync hooks — designed for local roots, mounted AWS S3 Files, or any replicated filesystem.

Mental model. Document files are authoritative. Indexes, event journals, and head pointers are derived state — rebuild them from documents if they ever drift.

Filesystem Layout

<root>/<collection>/
  .fylo/
    docs/
      4U/
        4UUB32VGUDW.json
    indexes/
      keys.wal
      keys.snapshot
    events/
      <collection>.ndjson

Installation

Install the published package directly from npm with Bun:

Bash

bun add @d31ma/fylo

A minimal setup — a root path and one collection:

TypeScript

import Fylo from '@d31ma/fylo'

const fylo = new Fylo({
  root: '/mnt/fylo'
})

await fylo.createCollection('users')

const id = await fylo.putData('users', {
  name: 'Ada',
  role: 'admin',
  tags: ['engineering', 'platform']
})

const doc = await fylo.getDoc('users', id).once()

Storage Model

Pass root to the constructor or set FYLO_ROOT in the environment. If neither is set, FYLO defaults to <cwd>/.fylo-data.

Configuration

# Preferred root config
FYLO_ROOT=/mnt/fylo

The index option controls the prefix index backend. Defaults to local filesystem (WAL + snapshot). Set to { backend: 's3-client', s3: { ... } } for a Bun S3Client-backed index. The worm option enables append-only mode with immutable versions and delete policies.

Environment Variables

Variable	Required	Details
FYLO_ROOT	Recommended	Filesystem root for collections. Defaults to <cwd>/.fylo-data.
FYLO_SCHEMA_DIR	Optional	Directory with per-collection schema manifests, version history, and upgrader modules.
FYLO_STRICT	Optional	When truthy, validates every write against the collection's head CHEX schema.
FYLO_ENCRYPTION_KEY	Conditional	Minimum 32 characters. Required when any schema declares $encrypted fields.
FYLO_CIPHER_SALT	Recommended	Unique per-deployment salt for PBKDF2 key derivation (100k iterations).
FYLO_LOGGING	Optional	Enables console logging of document writes and internal operations.

Default root. If no root is configured, FYLO stores data under .fylo-data in the current working directory.

NoSQL API

The collection API uses async iteration for reads, TTID-sequenced writes for documents, and query methods that compose across filtering, projection, and realtime streaming.

createCollection(name)

Create a collection directory with the FYLO metadata layout. Name must match /^[a-z0-9][a-z0-9\-]*[a-z0-9]$/.

dropCollection(name)

Remove a collection and all its documents, indexes, and event journals.

putData(collection, data)

Insert a document. Returns a generated TTID. Honors an existing TTID key in the data object.

batchPutData(collection, batch[])

Insert multiple documents concurrently (up to hardware concurrency). Returns an array of TTIDs.

getDoc(collection, id)

Returns an async iterable that yields the current doc then streams live updates. Call .once() for a one-shot read or .onDelete() to watch for deletion.

getLatest(collection, id)

Resolve a document or lineage ID to the current WORM head version. Essential in append-only mode.

getHistory(collection, id)

Return the full version chain for a document, newest first. Each entry includes the doc ID, version metadata, and timestamps.

findDocs(collection, query)

Index-accelerated query. Returns an async iterable with .collect() for materialization and .onDelete() for watching deletions in the result set.

patchDoc(collection, newDoc, oldDoc?)

Merge new fields into an existing document. Generates a new TTID version. Pass the previous doc for optimistic concurrency.

patchDocs(collection, update)

Find matching documents by query and patch each one. Returns the count of updated documents.

delDoc(collection, id)

Delete one document. In WORM tombstone mode, marks the head as deleted rather than removing the file.

delDocs(collection, query)

Delete all documents matching a query. Returns the count of deleted documents.

rebuildCollection(name)

Re-scan all document files to rebuild indexes and WORM head/version metadata. Returns a result with counts.

inspectCollection(name)

Report document count, index key count, and WORM head/version counts without scanning documents.

CRUD

const fylo = new Fylo({ root: '/mnt/fylo' })

const userId = await fylo.putData('users', {
  name: 'Jane Doe',
  age: 29,
  team: 'platform'
})

const doc = await fylo.getDoc('users', userId).once()

for await (const doc of fylo.findDocs('users', {
  $ops: [{ age: { $gte: 18 } }]
}).collect()) {
  console.log(doc)
}

SQL API

FYLO parses a SQL dialect through executeSQL(). Supports CREATE TABLE, DROP TABLE, SELECT, INSERT, UPDATE, and DELETE — translated into collection operations.

SQL

await fylo.executeSQL('CREATE TABLE posts')

await fylo.executeSQL(
  'INSERT INTO posts VALUES { "title": "Hello", "published": true }'
)

const posts = await fylo.executeSQL(
  'SELECT * FROM posts WHERE published = true'
)

Query Operators

The typed query surface supports comparison, pattern matching, array containment, and timestamp filtering. Multiple $ops entries form OR groups; conditions within a group are ANDed.

$eq $ne $gt $lt $gte $lte $like $contains

Operator	Required	Details
$eq / $ne	No	Exact match and exclusion for scalar values. Works on encrypted fields via HMAC blind indexes.
$gt / $lt / $gte / $lte	No	Numeric and timestamp range filtering. Not available on encrypted fields.
$like	No	SQL LIKE pattern matching with % wildcards. Uses trigram indexes for prefix and suffix patterns.
$contains	No	Array containment check. Matches when the field array includes the given value.
$created / $updated	No	Timestamp range filters derived from TTID-encoded document timestamps.

Query

// Find admins over 18 OR anyone on the platform team
const results = await fylo.findDocs('users', {
  $ops: [
    { role: { $eq: 'admin' }, age: { $gte: 18 } },
    { team: { $eq: 'platform' } }
  ],
  $limit: 50
}).collect()

Realtime Listeners

FYLO keeps an append-only event journal per collection. getDoc() and findDocs() return async iterables that first yield the current state then stream live changes from the journal.

Listeners

// Stream every admin doc, past and future
for await (const doc of fylo.findDocs('users', {
  $ops: [{ role: { $eq: 'admin' } }]
})) {
  console.log('admin changed:', doc)
}

// Watch for deletion of a specific document
for await (const deletedId of fylo.getDoc('users', userId).onDelete()) {
  console.log('deleted:', deletedId)
}

// One-shot read with .once()
const snapshot = await fylo.getDoc('users', userId).once()

Joins

joinDocs() supports inner, left, right, and outer cross-collection joins with rename, projection, grouping, and limit options.

JoinDocs

const joined = await fylo.joinDocs({
  $leftCollection: 'posts',
  $rightCollection: 'users',
  $mode: 'inner',
  $on: { userId: { $eq: 'id' } },
  $select: ['title', 'name'],
  $limit: 20
})

Bulk Import / Export

importBulkData() fetches JSON or JSONL from a URL with SSRF protection. exportBulkData() streams materialized documents as an async generator.

Bulk Data

const count = await fylo.importBulkData(
  'users',
  new URL('https://data.example.com/users.json'),
  {
    limit: 1000,
    maxBytes: 5 * 1024 * 1024,
    allowedHosts: ['data.example.com']
  }
)

for await (const doc of fylo.exportBulkData('users')) {
  console.log(doc)
}

Import guardrails. importBulkData() caps responses at 50 MiB by default. It rejects localhost, loopback, link-local, and private-network addresses unless explicitly opted in. DNS resolution is pinned for TLS ServerName verification.

Sync Hooks

FYLO does not ship its own cloud sync engine. Instead, attach hooks that run after local writes and deletes so your application can replicate the filesystem root however it wants.

Hooks

const fylo = new Fylo({
  root: '/mnt/fylo',
  syncMode: 'await-sync',
  sync: {
    async onWrite(event) {
      const file = Bun.file(event.path)
      await myS3Client.putObject({
        key: event.collection + '/' + event.docId + '.json',
        body: await file.arrayBuffer()
      })
    },
    async onDelete(event) {
      await myS3Client.deleteObject({
        key: event.collection + '/' + event.docId + '.json'
      })
    }
  }
})

Mode	Required	Details
await-sync	No	Default. Waits for the hook to complete and throws FyloSyncError if replication fails.
fire-and-forget	No	Commits locally first and runs the hook in the background.

Error shape. When an awaited sync fails after the local write succeeds, FYLO throws FyloSyncError with the collection, docId, path, and operation fields.

CLI

Three entry points — fylo.query, fylo.admin, and fylo.exec — share the same binary.

Shell

FYLO_ROOT=/tmp/fylo fylo.query "CREATE TABLE users"
FYLO_ROOT=/tmp/fylo fylo.query "SELECT * FROM users"
FYLO_ROOT=/tmp/fylo fylo.admin inspect users
FYLO_ROOT=/tmp/fylo fylo.admin rebuild users
FYLO_ROOT=/tmp/fylo fylo.admin get users 4UZVJ02N67O
FYLO_ROOT=/tmp/fylo fylo.admin latest users 4UZVJ02N67O
FYLO_ROOT=/tmp/fylo fylo.admin history users 4UZVJ02N67O
FYLO_ROOT=/tmp/fylo fylo.admin schema inspect users
FYLO_ROOT=/tmp/fylo fylo.admin schema doctor users
FYLO_ROOT=/tmp/fylo fylo.admin schema validate users < doc.json

Option	Details
--root	Override FYLO_ROOT for this invocation.
--schema-dir	Override FYLO_SCHEMA_DIR.
--json	Machine-readable JSON output instead of formatted tables.
--id-only	Return only the document ID from latest.
--page-size N	Repeat table headers every N rows.
--no-pager	Disable the pager (default: less -FRX, configured via FYLO_PAGER or PAGER).
--worm	Enable WORM-aware behavior (resolve heads, show version chains).

Machine Protocol

Use fylo.exec --request for programmatic access. Accepts JSON on stdin, a file path with @path, or a bare JSON string from -. Returns a versioned, structured envelope with timing.

Machine Request

echo '{
  "protocolVersion": 1,
  "requestId": "optional-correlation-id",
  "op": "findDocs",
  "collection": "users",
  "query": { "$ops": [{ "role": { "$eq": "admin" } }] }
}' | FYLO_ROOT=/tmp/fylo fylo.exec --request -

The protocol supports all 24 operations: executeSQL, createCollection, dropCollection, inspectCollection, rebuildCollection, getDoc, getLatest, getHistory, findDocs, joinDocs, putData, batchPutData, patchDoc, patchDocs, delDoc, delDocs, importBulkData, and the schema family: schemaInspect, schemaCurrent, schemaHistory, schemaDoctor, schemaValidate, schemaMaterialize.

Response format. Each reply includes protocolVersion, ok, op, requestId, durationMs, and either result or error (with name, message, code).

Query Builder

In addition to raw query objects, FYLO provides a chainable Parser.query() and Parser.join() API for building type-safe queries programmatically. Built queries can also be converted back to SQL via .toSQL().

Query Builder

import { Parser } from '@d31ma/fylo'

// Build a findDocs query
const q = Parser.query('users')
  .select('name', 'age')
  .where([{ role: { $eq: 'admin' } }])
  .limit(20)
  .build()
// → { $collection: 'users', $select: ['name', 'age'], $ops: [...], $limit: 20 }

// Build a join
const j = Parser.join('posts', 'users')
  .select('title', 'name')
  .innerJoin()
  .on({ userId: { $eq: 'id' } })
  .limit(50)
  .build()

// Round-trip to SQL
const sql = q.toSQL()
// → "SELECT name, age FROM users WHERE role = 'admin' LIMIT 20"

Local Queue

Enable a filesystem-based message queue with { queue: true } in the constructor. The LocalQueue provides at-least-once delivery with consumer-group checkpoints, advisory file locking, and a dead-letter queue — no external broker needed.

Feature	Details
Topics	Each topic is an append-only NDJSON file under .fylo/queue/topics/.
Consumer groups	Checkpoints stored per-group under .fylo/queue/consumers/.
Dead-letter queue	Messages that exhaust retries land in .fylo/queue/dlq/. Default max retries is 3.
Leases	Advisory file locks prevent concurrent processing of the same message.
Auto-ack	Messages are acknowledged on successful handler return. Set autoAck: false for manual ack() / nack().

Queue

const fylo = new Fylo({ root: '/mnt/fylo', queue: true })

// Publish a message to a topic
await fylo.queue.publish('email-welcome', {
  userId: '4UZVJ02N67O',
  email: 'ada@example.com'
})

// Drain and process pending messages
await fylo.queue.drain('email-welcome', {
  group: 'email-worker',
  maxRetries: 3
}, async (msg, ctx) => {
  await sendWelcomeEmail(msg.payload)
  // auto-acked on success; ctx.nack() to retry
})

Decorators

import { consume, publish } from '@d31ma/fylo'

class EmailWorker {
  @consume('email-welcome', { group: 'email-worker' })
  async handleWelcome(msg, ctx) {
    await sendWelcomeEmail(msg.payload)
  }

  @publish('email-welcome')
  async enqueueWelcome(user) {
    return { userId: user.id, email: user.email }
  }
}

Queue storage. All queue data lives under the FYLO root — no separate broker process. Messages are durable (written to disk) and survive process restarts.

Prefix Indexes

FYLO builds zero-payload prefix indexes to accelerate queries without duplicating document data. Each indexed field contributes key entries under .fylo/indexes/ that map field values to document IDs.

Index kind	Purpose	Key format
eq	Exact-match lookups ($eq, $ne)	field/eq/value/docId
n / nr	Numeric range scans ($gt, $lt, $gte, $lte)	Sortable numeric encoding; nr is reverse-order
f / r	Lexicographic range scans	Forward / reverse string ordering
g3	Trigram index for $like patterns	3-character substrings for prefix and suffix matching

Backend	Storage	Details
local-fs	WAL + snapshot	Default. Appends to keys.wal, compacts into keys.snapshot at 1 MB.
s3-client	Bun S3Client	Each collection maps to an S3 bucket. Keys are stored as zero-byte S3 objects.

S3 Index

const fylo = new Fylo({
  root: '/mnt/fylo',
  index: {
    backend: 's3-client',
    s3: {
      accessKeyId: process.env.S3_ACCESS_KEY,
      secretAccessKey: process.env.S3_SECRET_KEY,
      region: 'us-east-1',
      bucket: 'fylo-indexes'
    }
  }
})

Index rebuild. Call fylo.rebuildCollection('users') to re-scan all documents and regenerate the index. Useful after schema changes or if an index ever drifts from the document files.

WORM Mode

Enable append-only storage with { worm: { mode: 'append-only' } }. Every update writes a new immutable document file; old versions are preserved. Lineage heads and version metadata are tracked under .fylo/heads/ and .fylo/versions/.

Option	Values	Details
worm.mode	'off' (default), 'append-only'	When append-only, every write creates a new immutable version file.
worm.deletePolicy	'reject' (default), 'tombstone'	'reject' throws on any delete. 'tombstone' marks the head as deleted without removing files.

WORM

const fylo = new Fylo({
  root: '/mnt/fylo',
  worm: { mode: 'append-only', deletePolicy: 'tombstone' }
})

const lineageId = await fylo.putData('users', { name: 'Ada' })
const v2Id = await fylo.patchDoc('users', { name: 'Ada', role: 'admin' }, lineageId)

// Resolve to current head and inspect version chain
const head = await fylo.getLatest('users', lineageId)
const history = await fylo.getHistory('users', lineageId)

Compatibility Notes

This package no longer centers queued writes, Redis workers, or built-in migration commands. The following legacy APIs are removed or no-op:

Removed. rollback() is a no-op. getJobStatus(), getQueueStats(), replayDeadLetter(), and processQueuedWrites() throw an error explaining the feature was removed.

For local queuing, use the LocalQueue API (enable with { queue: true }). It provides at-least-once delivery with consumer-group checkpoints, advisory file locking, and a dead-letter queue — all filesystem-based.

Schema Validation

Set FYLO_SCHEMA_DIR to a directory of per-collection schema manifests. Enable FYLO_STRICT to validate every document against the collection's head CHEX schema on write.

Validation

FYLO_SCHEMA_DIR=/app/schemas
FYLO_STRICT=1

Each collection gets its own directory under FYLO_SCHEMA_DIR with a manifest.json tracking version history and optional upgraders/ modules for migrating documents between versions.

Schema Layout

schemas/users/
  manifest.json        # { current: "v2", versions: [...] }
  history/
    v1.json            # CHEX regex schema
    v2.json
  upgraders/
    v1-to-v2.js        # export default (doc) => upgradedDoc
  rules.json           # RLS rules (optional)

Field Encryption

Declare $encrypted fields in a collection's schema. FYLO encrypts those values with AES-256-GCM using random nonces. Exact-match queries use keyed HMAC-SHA256 blind indexes so plaintext never appears in document files, indexes, or event journals.

Schema

{
  "$encrypted": ["ssn", "accessToken"]
}

Key requirements. FYLO_ENCRYPTION_KEY must be at least 32 characters. Set FYLO_CIPHER_SALT to a unique per-deployment value. The key is derived via PBKDF2 with 100,000 iterations.

Query limitations. Only $eq and $ne operators work against encrypted fields (via blind indexes). Range operators and $like are not available on encrypted data.

Authorization (RLS)

FYLO enforces authorization through file-based Row-Level Security rules and scoped clients. Enable RLS in the constructor with { rls: true }, then call fylo.as(authContext) to get a scoped client that enforces per-collection rules.json files.

The scoped client has the same API shape as the root Fylo instance but every method is guarded: reads filter documents through RLS rules, writes check predicates, and unauthorized operations throw FyloAuthError.

RLS Rules (rules.json)

{
  "version": 1,
  "roles": [{
    "name": "admin",
    "apply_when": { "$in": ["admin", "%%user.roles"] },
    "read":   { "filter": {} },
    "insert": { "predicate": true },
    "update": { "filter": {}, "fields": null },
    "delete": { "filter": {} },
    "allow_actions": ["sql:execute", "bulk:export"]
  }, {
    "name": "user",
    "apply_when": { "$in": ["user", "%%user.roles"] },
    "read": { "filter": { "tenantId": { "$eq": "%%user.tenantId" } } },
    "insert": { "predicate": { "tenantId": { "$eq": "%%user.tenantId" } } },
    "update": { "filter": { "tenantId": { "$eq": "%%user.tenantId" } }, "fields": ["name", "email"] },
    "delete": { "filter": { "tenantId": { "$eq": "%%user.tenantId" } } }
  }]
}

Usage

const fylo = new Fylo({ root: '/mnt/fylo', rls: true })

const user = await verifyRequest(request)
const db = fylo.as({
  subjectId: user.id,
  tenantId: user.tenantId,
  roles: user.roles
})

// Scoped: only sees documents matching tenantId
for await (const doc of db.findDocs('users', {}).collect()) {
  console.log(doc)
}

Rules expression language. Supports $eq, $ne, $gt, $lt, $gte, $lte, $in, $and, $or, $not. Template variables %%user.*, %%root.*, and %%request.* are resolved from the auth context at call time.

Security

Concern	Guidance
Filesystem root	Point FYLO at a dedicated directory. Keep filesystem permissions tight around that root.
Secrets	Store FYLO_ENCRYPTION_KEY, FYLO_CIPHER_SALT, and sync credentials in environment variables or a secret manager.
App auth	Authenticate requests before FYLO. Use fylo.as(authContext) with file-based RLS rules — the scoped client fails closed.
Replication hooks	Treat sync.onWrite and sync.onDelete as production data movement code. Add retries, logging, and alerting.
Schema validation	Use FYLO_SCHEMA_DIR and FYLO_STRICT for shape guarantees at write time.
Import safety	SSRF guard rejects private-network URLs by default. Use allowedHosts and allowPrivateNetwork explicitly when importing from internal sources.
WORM integrity	In append-only mode with deletePolicy: 'reject', data is immutable after write — no code path can delete or overwrite stored documents.

Local Development

No AWS required. Point the root at a temp directory and FYLO creates its document, index, and event files locally. When ready for production, mount S3 Files or another replicated filesystem at the same path.

Local Root

mkdir -p /tmp/fylo
export FYLO_ROOT=/tmp/fylo

bun run app.ts

For local queuing during development, enable { queue: true } in the constructor. The LocalQueue is filesystem-based and needs no external broker.