Skip to content
CLUB

Storage Backends

club uses three independent storage layers, each behind an abstract interface. You can mix and match backends freely — for example, SQLite for metadata with S3 for blob storage.

Three Storage Layers

LayerPurposeDefaultAlternatives
Metadata StoreRelational data (packages, versions, users, tokens, publishers, audit log)SQLitePostgreSQL
Blob StorePackage tarballs, per-version screenshots, and optionally dartdoc indexed blobsFilesystemS3-compatible, GCS (native)
Search IndexFull-text package searchSQLite FTS5Meilisearch

Each layer is configured independently via a single environment variable:

Terminal window
DB_BACKEND=sqlite          # or postgres
BLOB_BACKEND=filesystem    # or s3 | gcs
SEARCH_BACKEND=sqlite      # or meilisearch

On top of these three core layers there’s one serve-strategy switch that controls where rendered dartdoc HTML comes from at request time:

Terminal window
DARTDOC_BACKEND=filesystem # or blob — orthogonal to BLOB_BACKEND

DARTDOC_BACKEND=filesystem serves HTML trees from a local directory (DARTDOC_PATH). DARTDOC_BACKEND=blob persists a per-package indexed blob via the configured Blob Store — same one used for tarballs — and serves via byte-range reads. See Dartdoc Serving Specifications.

See Environment Variables for the full list of keys each backend accepts, and Data Directory Layout for the full /data tree showing where each layer writes on disk.

Metadata Store: SQLite vs PostgreSQL

SQLite is the default metadata store. It requires zero external dependencies and stores everything in a single file.

Pros:

  • Zero setup — works out of the box
  • Single-file database, easy to backup and restore
  • WAL mode enables concurrent reads during writes
  • Excellent performance for small to medium deployments (hundreds of packages)
  • No external process to manage

Cons:

  • Single-writer model (one write at a time, with WAL queuing)
  • Not ideal for high-concurrency write workloads
  • No built-in replication

Configuration:

Terminal window
DB_BACKEND=sqlite
SQLITE_PATH=/data/db/club.db

SQLite pragmas set on every connection:

PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
PRAGMA busy_timeout = 5000;

When to Choose PostgreSQL

Consider switching from SQLite to PostgreSQL when:

  • You have more than 10 concurrent publishing users
  • You need database replication or failover
  • You are running multiple club server instances behind a load balancer
  • Your package catalog exceeds 1,000 packages with frequent updates

Blob Store: Filesystem vs S3 vs GCS

Stores package tarballs as files on the local filesystem.

Pros:

  • Zero setup
  • Fast local reads (no network latency)
  • Easy to backup with standard tools (tar, rsync)
  • Simple to inspect and debug

Cons:

  • Tied to a single server’s disk
  • Disk space limited by the host
  • No built-in redundancy
  • Cannot be shared across multiple server instances

Configuration:

Terminal window
BLOB_BACKEND=filesystem
BLOB_PATH=/data/blobs

File layout:

/data/blobs/
├── my_package/
│   ├── 1.0.0/
│   │   ├── artifacts/
│   │   │   └── package.tar.gz
│   │   └── screenshots/
│   │       ├── 0
│   │       └── 1
│   └── 2.0.0/
│       └── artifacts/
│           └── package.tar.gz
├── my_package/dartdoc/latest/     # only when DARTDOC_BACKEND=blob
│   ├── blob
│   └── index.json
└── other_package/
    └── 0.1.0/
        └── artifacts/
            └── package.tar.gz

Files are written atomically using a temp-file-then-rename pattern to prevent partial writes. Per-version screenshots and blob-mode dartdoc live under the same package directory so the whole blob tier can be backed up as one unit.

When to Choose S3

Consider switching from filesystem to S3 when:

  • You need more storage than a single disk provides
  • You are running multiple club instances behind a load balancer
  • You want built-in redundancy without managing RAID or replication
  • You are already using AWS or a cloud provider with S3-compatible storage

Search Index: SQLite FTS5 vs Meilisearch

Uses SQLite’s built-in FTS5 full-text search engine. Runs in the same process as the server with no external dependencies.

Pros:

  • Zero setup — built into SQLite
  • Fast for small to medium catalogs
  • No external service to manage
  • Supports prefix matching and tag filters

Cons:

  • Limited ranking/relevance tuning
  • No typo tolerance
  • Performance degrades with very large catalogs (10,000+ packages)

Configuration:

Terminal window
SEARCH_BACKEND=sqlite

The FTS5 index is maintained by the search-index layer, which updates the index whenever packages are created, updated, or deleted.

When to Choose Meilisearch

Consider switching from SQLite FTS5 to Meilisearch when:

  • You have a large package catalog (1,000+ packages)
  • You want typo-tolerant search (e.g., “htttp” still finds “http”)
  • You need advanced faceting or filtering beyond what FTS5 provides

Switching Backends

Switching the Metadata Store

To switch from SQLite to PostgreSQL:

  1. Set up a PostgreSQL database
  2. Update your configuration:
    Terminal window
    DB_BACKEND=postgres
    POSTGRES_URL=postgres://club:secret@db.example.com:5432/club
  3. Restart the server — migrations run automatically on startup and create the schema

Switching the Blob Store

To switch from filesystem to S3:

  1. Set up an S3 bucket (or MinIO instance)
  2. Copy existing tarballs to the S3 bucket preserving the path structure:
    Terminal window
    aws s3 sync /data/blobs/ s3://club-packages/
  3. Update your configuration:
    Terminal window
    BLOB_BACKEND=s3
    S3_BUCKET=club-packages
    S3_REGION=us-east-1
    S3_ACCESS_KEY=...
    S3_SECRET_KEY=...
  4. Restart the server

Switching the Search Backend

To switch from SQLite FTS5 to Meilisearch:

  1. Start a Meilisearch instance
  2. Update your configuration:
    Terminal window
    SEARCH_BACKEND=meilisearch
    MEILISEARCH_URL=http://meilisearch:7700
    MEILISEARCH_KEY=your-api-key
  3. Restart the server — the search index is rebuilt automatically from the metadata store

Small Team (up to 50 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=filesystem
SEARCH_BACKEND=sqlite

Zero external dependencies. Everything runs in a single container.

Medium Team (50-500 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=s3
SEARCH_BACKEND=sqlite

Offload blob storage to S3 for durability and shared access, but keep SQLite for simplicity.

Large Organization (500+ packages)

Terminal window
DB_BACKEND=postgres
BLOB_BACKEND=s3
SEARCH_BACKEND=meilisearch

Full external stack for maximum scalability, concurrency, and search quality.