Storage Configuration

Storage configuration determines where Emend stores data.

Default Storage (Open Source)

The open-source version uses SQLite by default — no storage configuration is needed. Emend automatically creates and manages a local SQLite database, so you can start using it immediately without any setup.

# No storage configuration required — SQLite is used automaticallyclient = EmendClient()  # see Quickstart for connection optionsconfig = client.get_config()# config.storage_config is already set to SQLite

Supabase Storage

For production deployments that need managed cloud storage, you can configure Supabase manually:

from emend.models.config_schema import StorageConfigSupabase storage = StorageConfigSupabase(    url="https://your-project.supabase.co",    key="your_service_role_key",    db_url="postgresql://emend_user:replace-me@host:5432/postgres",    # Optional reader credentials for search traffic. Search reads use these    # credentials; writes and consistency-sensitive reads still use url/key.    read_url="https://your-reader.supabase.co",    read_key="your_reader_service_role_key",)config.storage_config = storageclient.set_config(config)

PostgreSQL Storage

from emend.models.config_schema import StorageConfigPostgres storage = StorageConfigPostgres(    db_url="postgresql://emend_user:replace-me@host:5432/database",    schema="public",    # Optional reader pool for search traffic    read_db_url="postgresql://emend_reader:replace-me@reader:5432/database",    read_pool_size=10,)config.storage_config = storageclient.set_config(config)

The pool is shared by all concurrent work for an organization. When in-flight queries exceed pool_size, additional queries queue for up to pool_acquire_timeout seconds rather than failing immediately; only a query that waits longer than the timeout raises an error. Because pool_size is per organization, the total server-side connection count scales with the number of active organizations — size it against your database's max_connections.

In self-host deployments, pool settings can be set without editing config via EMEND_POSTGRES_POOL_SIZE, EMEND_POSTGRES_POOL_ACQUIRE_TIMEOUT, EMEND_POSTGRES_READ_DB_URL, EMEND_POSTGRES_READ_POOL_SIZE, and EMEND_POSTGRES_READ_POOL_ACQUIRE_TIMEOUT.

Search/read timeout knobs are also available for production deployments: EMEND_SUPABASE_HTTP_TIMEOUT_SECONDS for Supabase/PostgREST and EMEND_POSTGRES_STATEMENT_TIMEOUT_MS for native Postgres.

Config Encryption

Generate a key with:

uv run python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Row Retention

Emend applies high-water row limits to data tables on the publish path. When an eligible table reaches its limit, the server deletes the oldest 20% of rows for that table by its target-specific ordering column; not every target uses created_at.

# Defaults to 250000 rows per targetEMEND_ROW_LIMIT_INTERACTIONS=500000EMEND_ROW_LIMIT_PROFILES=250000 # Set a target to 0 to disable its cleanupEMEND_ROW_LIMIT_PLAYBOOK_OPTIMIZATION_EVENTS=0 # Defaults to 300 seconds. Set to 0 or a negative value to disable the periodic# cleanup sweep.EMEND_RETENTION_CLEANUP_INTERVAL_SECONDS=300

The legacy INTERACTION_CLEANUP_THRESHOLD variable still applies to interactions when EMEND_ROW_LIMIT_INTERACTIONS is unset.

The user_playbook_exposure_events target is fixed at 250,000 rows in code and does not accept a EMEND_ROW_LIMIT_USER_PLAYBOOK_EXPOSURE_EVENTS override.

Most retention targets are ordered by created_at. The exceptions are: offline_tuner_reward_label rows use label_created_at, and user_playbook_exposure_events rows use ingested_at. Exposure events also have a 14-day minimum age floor, so row-count cleanup cannot delete them until they are at least 14 days old.

Enterprise Self-Host Single Database

For Supabase self-host, configure DATA_SUPABASE_URL, DATA_SUPABASE_KEY, and DATA_DB_URL. For vanilla Postgres self-host, configure DATA_DB_URL. Startup applies auth and data migrations to that same database and stores the generated configuration_json in public.organizations.

Usage-metering WAL volume (self-host)

In required mode, usage is written to a small encrypted write-ahead log (WAL) on disk rather than your database. Mount a persistent, writable volume at EMEND_USAGE_WAL_PATH (default ~/.emend/usage_wal) for restart-safe metering. Boot fails when the resolved directory is not writable; an ephemeral or default path logs a warning.

If the control plane temporarily cannot complete a usage-ingest cardinality read, it returns HTTP 503 with Retry-After: 5. The shipper retains the batch in its WAL and retries without double-counting acknowledged usage. Keep the WAL volume intact while database availability recovers.

For multi-instance deployments, configure one shared persistent mount root and let startup create derived per-instance directories at <root>/<instance-id>. Distinct per-replica persistent volumes are an alternative when every replica receives a stable, unique path. Emend's generated, file-backed instance id is preferred; set EMEND_INSTANCE_ID only when the orchestrator guarantees uniqueness. If required-mode WAL files become unreadable, recover the original preserved files rather than starting from an empty directory.

In exempt mode, Emend does not construct or validate the usage WAL, and no WAL path, persistence, or mount is required. Early startup may still normalize or derive a configured path environment value before selecting the exempt composition.

Activating a Self-Host Data Plane

# The activation key shown once in the Emend admin portal.BYOC_DEPLOYMENT_SECRET="rflx-dep-…"# Optional — the Emend control-plane base URL. Defaults to the Emend cloud# control plane (https://www.emend.online); set it only to target a different one.# CONTROL_PLANE_URL="https://www.emend.online"

CONTROL_PLANE_URL is the canonical setting. Older deployments that still set CONTROL_PLANE_INGEST_URL continue to work as a fallback, but new deployments should use CONTROL_PLANE_URL.

On startup the data plane calls POST /api/billing/byoc/activate with the activation key in the X-Deployment-Secret header. The control plane confirms the key→deployment binding and returns your deployment id, org id, and central public key.

After activation, the data plane ships usage and pulls its entitlement lease automatically. You do not need to set BYOC_DEPLOYMENT_ID or CONTROL_PLANE_PUBLIC_KEY; the activation handshake supplies them.