Skip to content
CLUB

Storage Backends

club uses three independent storage layers, each behind an abstract interface. You can mix and match backends freely — for example, SQLite for metadata with S3 for blob storage.

Three Storage Layers

LayerPurposeDefaultAlternatives
Metadata StoreRelational data (packages, versions, users, tokens, publishers, audit log)SQLitePostgreSQL
Blob StorePackage tarballs, per-version screenshots, and optionally dartdoc indexed blobsFilesystemS3-compatible, GCS (native)
Search IndexFull-text package searchSQLite FTS5Meilisearch

Each layer is configured independently via a single environment variable:

Terminal window
DB_BACKEND=sqlite          # or postgres
BLOB_BACKEND=filesystem    # or s3 | gcs
SEARCH_BACKEND=sqlite      # or meilisearch

On top of these three core layers there’s one serve-strategy switch that controls where rendered dartdoc HTML comes from at request time:

Terminal window
DARTDOC_BACKEND=filesystem # or blob — orthogonal to BLOB_BACKEND

DARTDOC_BACKEND=filesystem serves HTML trees from a local directory (DARTDOC_PATH). DARTDOC_BACKEND=blob persists a per-package indexed blob via the configured Blob Store — same one used for tarballs — and serves via byte-range reads. See Dartdoc Serving Specifications.

See Environment Variables for the full list of keys each backend accepts, and Data Directory Layout for the full /data tree showing where each layer writes on disk.

Metadata Store: SQLite vs PostgreSQL

SQLite is the default metadata store. It requires zero external dependencies and stores everything in a single file.

Pros:

  • Zero setup — works out of the box
  • Single-file database, easy to backup and restore
  • WAL mode enables concurrent reads during writes
  • Excellent performance for small to medium deployments (hundreds of packages)
  • No external process to manage

Cons:

  • Single-writer model (one write at a time, with WAL queuing)
  • Not ideal for high-concurrency write workloads
  • No built-in replication

Configuration:

Terminal window
DB_BACKEND=sqlite
SQLITE_PATH=/data/db/club.db

SQLite pragmas set on every connection:

PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
PRAGMA busy_timeout = 5000;

When to Choose PostgreSQL

Consider switching from SQLite to PostgreSQL when:

  • You have more than 10 concurrent publishing users
  • You need database replication or failover
  • You are running multiple club server instances behind a load balancer
  • Your package catalog exceeds 1,000 packages with frequent updates

Blob Store: Filesystem vs S3 vs GCS

Stores package tarballs as files on the local filesystem.

Pros:

  • Zero setup
  • Fast local reads (no network latency)
  • Easy to backup with standard tools (tar, rsync)
  • Simple to inspect and debug

Cons:

  • Tied to a single server’s disk
  • Disk space limited by the host
  • No built-in redundancy
  • Cannot be shared across multiple server instances

Configuration:

Terminal window
BLOB_BACKEND=filesystem
BLOB_PATH=/data/blobs

File layout:

/data/blobs/
├── my_package/
│   ├── 1.0.0/
│   │   ├── artifacts/
│   │   │   └── package.tar.gz
│   │   └── screenshots/
│   │       ├── 0
│   │       └── 1
│   └── 2.0.0/
│       └── artifacts/
│           └── package.tar.gz
├── my_package/dartdoc/latest/     # only when DARTDOC_BACKEND=blob
│   ├── blob
│   └── index.json
└── other_package/
    └── 0.1.0/
        └── artifacts/
            └── package.tar.gz

Files are written atomically using a temp-file-then-rename pattern to prevent partial writes. Per-version screenshots and blob-mode dartdoc live under the same package directory so the whole blob tier can be backed up as one unit.

When to Choose S3

Consider switching from filesystem to S3 when:

  • You need more storage than a single disk provides
  • You are running multiple club instances behind a load balancer
  • You want built-in redundancy without managing RAID or replication
  • You are already using AWS or a cloud provider with S3-compatible storage

Search Index: SQLite FTS5 vs Meilisearch

Uses SQLite’s built-in FTS5 full-text search engine. Runs in the same process as the server with no external dependencies.

Pros:

  • Zero setup — built into SQLite
  • Fast for small to medium catalogs
  • No external service to manage
  • Supports prefix matching and tag filters

Cons:

  • Limited ranking/relevance tuning
  • No typo tolerance
  • Performance degrades with very large catalogs (10,000+ packages)

Configuration:

Terminal window
SEARCH_BACKEND=sqlite

The FTS5 index is maintained by the search-index layer, which updates the index whenever packages are created, updated, or deleted.

When to Choose Meilisearch

Consider switching from SQLite FTS5 to Meilisearch when:

  • You have a large package catalog (1,000+ packages)
  • You want typo-tolerant search (e.g., “htttp” still finds “http”)
  • You need advanced faceting or filtering beyond what FTS5 provides

Switching Backends

Switching the Metadata Store

To switch from SQLite to PostgreSQL:

  1. Set up a PostgreSQL database
  2. Update your configuration:
    Terminal window
    DB_BACKEND=postgres
    POSTGRES_URL=postgres://club:secret@db.example.com:5432/club
  3. Restart the server — migrations run automatically on startup and create the schema

Switching the Blob Store

To switch from filesystem to S3:

  1. Set up an S3 bucket (or MinIO instance)
  2. Copy existing tarballs to the S3 bucket preserving the path structure:
    Terminal window
    aws s3 sync /data/blobs/ s3://club-packages/
  3. Update your configuration:
    Terminal window
    BLOB_BACKEND=s3
    S3_BUCKET=club-packages
    S3_REGION=us-east-1
    S3_ACCESS_KEY=...
    S3_SECRET_KEY=...
  4. Restart the server

Switching the Search Backend

To switch from SQLite FTS5 to Meilisearch:

  1. Start a Meilisearch instance
  2. Update your configuration:
    Terminal window
    SEARCH_BACKEND=meilisearch
    MEILISEARCH_URL=http://meilisearch:7700
    MEILISEARCH_KEY=your-api-key
  3. Restart the server — the search index is rebuilt automatically from the metadata store

Small Team (up to 50 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=filesystem
SEARCH_BACKEND=sqlite

Zero external dependencies. Everything runs in a single container.

Medium Team (50-500 packages)

Terminal window
DB_BACKEND=sqlite
BLOB_BACKEND=s3
SEARCH_BACKEND=sqlite

Offload blob storage to S3 for durability and shared access, but keep SQLite for simplicity.

Large Organization (500+ packages)

Terminal window
DB_BACKEND=postgres
BLOB_BACKEND=s3
SEARCH_BACKEND=meilisearch

Full external stack for maximum scalability, concurrency, and search quality.

Static-site archives

Package sites use local filesystem storage independently of BLOB_BACKEND. This applies even when package blobs use S3 or GCS. The default root is /data/sites, with one active set per package:

sites/<package>/<site>/
  site.tar.gz     # Archived target
  redirect.json   # URL target, instead of site.tar.gz
  metadata.json   # Optional display label

Keep SITES_PATH on a persistent volume and include it in backups alongside the database and package blobs. Archive and URL-target metadata are primary data, not regenerable caches. Use a single server process for this directory; sharing it between multiple replicas is not supported. Leave disk headroom for uploads, the new site set, and the previous set during replacement. Stop publishing while backing up or restoring the directory so that a replacement cannot race the copy.

The server validates archives without executing their content. It stages a complete replacement, restores the previous set on promotion failure, and recovers interrupted promotions at startup. Files are unpacked for rendering in the browser, not served as extracted HTML from the API host.

Environment variableYAML keyDefault
SITES_PATHsites_path/data/sites
MAX_SITESmax_sites20
MAX_SITE_ARCHIVE_BYTESmax_site_archive_bytes104857600 (100 MiB)
MAX_SITES_TOTAL_BYTESmax_sites_total_bytes524288000 (500 MiB)
MAX_SITE_EXPANDED_BYTESmax_site_expanded_bytes524288000 (500 MiB, including tar headers and padding)
MAX_SITE_ENTRIESmax_site_entries10000

The package archive retains its separate MAX_UPLOAD_BYTES budget. Site manifests are limited to 64 KiB. URL targets count toward the site count and manifest limit, not archive-byte budgets. The CLI also enforces the default preparation budgets; the browser caps an archive at 100 MiB compressed, 500 MiB expanded, and 10,000 tar entries. Increasing server limits does not lift these client caps.

Browser previews

As of 0.8.1, archived previews work on the existing Club hostname. No additional DNS, subdomains, exposed ports, or runner configuration are required. Club serves only its bundled runtime at /site-runner/; uploaded sites are still delivered as authenticated archives and extracted in the browser.

The runtime HTML carries an HTTP CSP sandbox without allow-same-origin, so both framed and direct visits have an opaque origin. Preserve that response’s Content-Security-Policy in your reverse proxy. Do not apply the main dashboard’s restrictive script/connect CSP to /site-runner/: the sandboxed preview needs Blob resources and external site assets. Runtime scripts have credential-free CORS access; this does not change CORS or authentication on the archive API.

Environment variableYAML keyDefault
SITE_RUNNER_URLsite_runner_urlUnset, use the bundled same-host runner
SITE_RUNNER_PORTsite_runner_port8081, used only with an explicit runner URL

Existing separate-runner deployments remain supported. If explicitly set, SITE_RUNNER_URL must be a separate HTTPS origin (HTTP is permitted only on loopback), with no credentials, query, fragment, or path other than /. The listener port must differ from LISTEN_PORT. Route that hostname only to the runner listener; do not share Club cookies or grant credentialed API CORS. An unset URL starts no extra listener and uses the main Club port instead. Restart the server after changing these optional settings.

Keep /data on a persistent volume. If SITES_PATH is outside /data, mount it too and make it writable by the container’s user. Never expose /data/sites or serve extracted uploaded files through a static web server.

Authentication and cache behavior

The Club page requests /api/packages/<package>/sites and, for an archive target, /api/packages/<package>/sites/<site>/archive through the normal access checks. Private packages require authentication. Anonymous site reads are allowed only for public packages when public packages are enabled. The runner receives site content, not Club credentials, and provides no authenticated API proxy.

The Club page caches compressed downloads in browser Cache Storage, separated by signed-in user (or anonymous browsing), and revalidates each opening using the archive’s SHA-256 ETag. An unchanged archive returns 304; changed bytes return 200 and replace the local copy after verification. Access and Enable sites checks happen before conditional responses. Preserve ETag and Cache-Control: private, no-cache for archive responses; do not place private archives in a shared CDN cache or strip their access checks.

The runner unpacks the archive and passes files to an inner iframe with allow-scripts allow-forms, without allow-same-origin. The uploaded site’s origin is opaque, isolating it from Club’s DOM and origin storage and from other preview frames. The outer runtime and uploaded content both have opaque origins, including when the runtime is served on the Club hostname. Extracted files stay in memory, and no Service Worker serves them. See preview compatibility before choosing an archived target for an application.

If you used an earlier Service-Worker-based preview prototype, clear that old runner origin’s browser site data once to remove its workers and extracted caches. The opaque runtime cannot access or delete origin storage, and does not use those old copies. Ordinary 0.8.0 opaque previews do not create extracted caches.

Cached downloads are subject to browser eviction, not guaranteed offline availability. They are also downloaded copies, not remotely revocable data. Already-open previews can continue running after an access change. The Enable sites checkbox, unchecked by default, blocks new site reads and Club redirects when unchecked without deleting sites or stopping publishes. External websites remain available at their own URLs.