Skip to content
CLUB

Storage Backends

club uses three independent storage layers, each behind an abstract interface. You can mix and match backends freely — for example, SQLite for metadata with S3 for blob storage.

Three Storage Layers

LayerPurposeDefaultAlternatives
Metadata StoreRelational data (packages, versions, users, tokens, publishers, audit log)SQLitePostgreSQL
Blob StoreBinary .tar.gz package archivesFilesystemS3-compatible, GCS (native)
Search IndexFull-text package searchSQLite FTS5Meilisearch

Each layer is configured independently via a single environment variable:

Terminal window
DB_BACKEND=sqlite          # or postgres
BLOB_BACKEND=filesystem    # or s3 | gcs
SEARCH_BACKEND=sqlite      # or meilisearch

See Environment Variables for the full list of keys each backend accepts.

Metadata Store: SQLite vs PostgreSQL

SQLite is the default metadata store. It requires zero external dependencies and stores everything in a single file.

Pros:

  • Zero setup — works out of the box
  • Single-file database, easy to backup and restore
  • WAL mode enables concurrent reads during writes
  • Excellent performance for small to medium deployments (hundreds of packages)
  • No external process to manage

Cons:

  • Single-writer model (one write at a time, with WAL queuing)
  • Not ideal for high-concurrency write workloads
  • No built-in replication

Configuration:

Terminal window
DB_BACKEND=sqlite
SQLITE_PATH=/data/club.db

SQLite pragmas set on every connection:

PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA foreign_keys = ON;
PRAGMA busy_timeout = 5000;

When to Choose PostgreSQL

Consider switching from SQLite to PostgreSQL when:

  • You have more than 10 concurrent publishing users
  • You need database replication or failover
  • You are running multiple club server instances behind a load balancer
  • Your package catalog exceeds 1,000 packages with frequent updates

Blob Store: Filesystem vs S3 vs GCS

Stores package tarballs as files on the local filesystem.

Pros:

  • Zero setup
  • Fast local reads (no network latency)
  • Easy to backup with standard tools (tar, rsync)
  • Simple to inspect and debug

Cons:

  • Tied to a single server’s disk
  • Disk space limited by the host
  • No built-in redundancy
  • Cannot be shared across multiple server instances

Configuration:

Terminal window
BLOB_BACKEND=filesystem
BLOB_PATH=/data/packages

File layout:

/data/packages/
├── my_package/
│   ├── 1.0.0.tar.gz
│   ├── 1.1.0.tar.gz
│   └── 2.0.0.tar.gz
└── other_package/
    └── 0.1.0.tar.gz

Files are written atomically using a temp-file-then-rename pattern to prevent partial writes.

When to Choose S3

Consider switching from filesystem to S3 when:

  • You need more storage than a single disk provides
  • You are running multiple club instances behind a load balancer
  • You want built-in redundancy without managing RAID or replication
  • You are already using AWS or a cloud provider with S3-compatible storage

Search Index: SQLite FTS5 vs Meilisearch

Uses SQLite’s built-in FTS5 full-text search engine. Runs in the same process as the server with no external dependencies.

Pros:

  • Zero setup — built into SQLite
  • Fast for small to medium catalogs
  • No external service to manage
  • Supports prefix matching and tag filters

Cons:

  • Limited ranking/relevance tuning
  • No typo tolerance
  • Performance degrades with very large catalogs (10,000+ packages)

Configuration:

Terminal window
SEARCH_BACKEND=sqlite

The FTS5 index is maintained automatically via database triggers whenever packages are created, updated, or deleted.

When to Choose Meilisearch

Consider switching from SQLite FTS5 to Meilisearch when:

  • You have a large package catalog (1,000+ packages)
  • You want typo-tolerant search (e.g., “htttp” still finds “http”)
  • You need advanced faceting or filtering beyond what FTS5 provides

Switching Backends

Switching the Metadata Store

To switch from SQLite to PostgreSQL:

  1. Set up a PostgreSQL database
  2. Update your configuration:
    Terminal window
    DB_BACKEND=postgres
    POSTGRES_URL=postgres://club:secret@db.example.com:5432/club
  3. Restart the server — migrations run automatically on startup and create the schema

Switching the Blob Store

To switch from filesystem to S3:

  1. Set up an S3 bucket (or MinIO instance)
  2. Copy existing tarballs to the S3 bucket preserving the path structure:
    Terminal window
    aws s3 sync /data/packages/ s3://club-packages/
  3. Update your configuration:
    Terminal window
    BLOB_BACKEND=s3
    S3_BUCKET=club-packages
    S3_REGION=us-east-1
    S3_ACCESS_KEY=...
    S3_SECRET_KEY=...
  4. Restart the server

Switching the Search Backend

To switch from SQLite FTS5 to Meilisearch:

  1. Start a Meilisearch instance
  2. Update your configuration:
    Terminal window
    SEARCH_BACKEND=meilisearch
    MEILISEARCH_URL=http://meilisearch:7700
    MEILISEARCH_KEY=your-api-key
  3. Restart the server — the search index is rebuilt automatically from the metadata store

Small Team (up to 50 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=filesystem
SEARCH_BACKEND=sqlite

Zero external dependencies. Everything runs in a single container.

Medium Team (50-500 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=s3
SEARCH_BACKEND=sqlite

Offload blob storage to S3 for durability and shared access, but keep SQLite for simplicity.

Large Organization (500+ packages)

Terminal window
DB_BACKEND=postgres
BLOB_BACKEND=s3
SEARCH_BACKEND=meilisearch

Full external stack for maximum scalability, concurrency, and search quality.