Monitoring and backups
What to log, watch and back up once OpenDash is running, and how to scale it out.
Logs and traces
Both processes log JSON lines to stdout, so any log collector that reads container output works.
To send traces to an OpenTelemetry collector, set OTEL_EXPORTER_OTLP_ENDPOINT (OTLP over HTTP, e.g. http://127.0.0.1:4318) and optionally OTEL_SERVICE_NAME. Spans carry ids, routes, timings and usage. They never carry a question, its SQL, an answer, or a URL's query string.
What to watch
| Signal | Why it matters |
|---|---|
| Queue depth and dead-letter queue size | Runs and deliveries are jobs. A growing queue means workers can't keep up. |
| Runs still active past their time budget | A stuck run, a slow source or a slow provider. |
| Missing future partitions of the high-volume tables | Correctness-critical. Alert on it. |
pg_notification_queue_usage() above 10% | Change notifications are backing up. |
| Postgres connection counts | Each process holds one LISTEN connection plus its pools. |
| Query failures per source | Credentials, network or a source under load. |
Backups
Back up two things, separately:
- PostgreSQL. Managed point-in-time recovery, or
pg_dumpplus WAL archiving (for example WAL-G). OPENDASH_KEK, offline. It is the only way to decrypt the connection passwords and model keys stored in Postgres. Losing it means re-entering every one of them.
Your analytical data stays in your own databases, so it isn't part of OpenDash's backup.
Scaling
The same images scale out without a migration:
Move to a managed Postgres
Point DATABASE_URL, DATABASE_DIRECT_URL and DATABASE_QUEUE_URL (and the auth and system URLs) at it. See PostgreSQL for which connection may use a pooler.
Add web replicas
web is stateless. Each replica holds one LISTEN connection.
Split the workers
Run the worker image several times with different WORKER_POOLS:
WORKER_POOLS=interactive # agent runs and what they send, such as Slack replies; dashboard summaries
WORKER_POOLS=query # connection checks, dashboard queries, metric checks, routine runs and sends
WORKER_POOLS=sync # learning sources, testing definitionsScale interactive with the number of people asking questions.