The dbt daily run failed¶

The daily run is a Kubernetes CronJob at 06:00 UTC, ~3h40m, with concurrencyPolicy: Forbid, backoffLimit: 0 and restartPolicy: Never — one attempt, no Kubernetes retries. That is deliberate: the orchestrator owns retries internally.

It runs cron_preview.sh → scripts/run_dbt_observability.sh, whose only mandatory step is dbt-run. source-freshness failing every night is expected and non-mandatory.

Separately, a live loop Deployment runs dbt run --select tag:live (19 models) every 45 s, and a static server Deployment serves the published logs and metrics.

1. Read the failure¶

Find the failed job's pod and grep its log:

MANDATORY STEP FAILED | ] Failed: | Code: [0-9]+ | REFUSED | TRANSIENT= | PERMANENT=

Only three job records are kept per outcome, so a pod older than ~3 runs is gone. After that, the published artifacts are served by the static server under /logs/ and /reports/ — reach it with a port-forward.

The orchestrator's step summary, the TRANSIENT= / PERMANENT= lines and the retry outcome go to stdout only — they are not in the published dbt.log.

2. Classify¶

Transient: Code: 159 | 209 | 210 | 394, QUERY_WAS_CANCELLED, TIMEOUT_EXCEEDED, SOCKET_TIMEOUT, NETWORK_ERROR, SSLError, UNEXPECTED_EOF_WHILE_READING, HTTPSConnectionPool, RemoteDisconnected, ConnectionResetError, Broken pipe.

Code: 341 on a daily run means a mutation OOMed — the model needs the microbatch path, not a retry. Code: 388 means the merge pool is full; wait, do not loop-retry.

The cron already retried transients once at --threads 1, so what you see failed twice.

3. Is it failed, or still running?¶

A job past ~3h40m may be stalled rather than failed: check whether it is still active, whether the last log line is moving, and its resource usage.

SELECT * FROM system.mutations WHERE is_done = 0 AND is_killed = 0 ORDER BY create_time DESC LIMIT 20;

4. Rerun only what failed¶

The failed node ids are in the per-batch stash:

python /app/scripts/refresh/classify_failed_nodes.py --stash-dir /app/target/failed_batches
# prints TRANSIENT=<ids> and PERMANENT=<ids>
dbt run --select ⟨node⟩+ --threads 1

Run these inside a one-shot dbt job (how). The flag is --stash-dir, not --dir.

5. Refused stages¶

grep REFUSED

A refusal means the model's watermark is more than --max-slices-per-stage (30) days behind, so the daily runner declined to catch it up one day at a time. Three causes, three answers:

Finding consumers — the manifest closure, plus the two surfaces dbt does not know about:

dbt ls -s ⟨model⟩+ --resource-type model --output name
grep -rn "⟨model⟩" semantic/                          # semantic layer
grep -rn "dbt\.⟨model⟩" ../metrics-dashboard/src/queries/   # dashboard SQL

Stalled vs dormant — compare the table's max against the chain. *_events models read execution.logs by address; *_calls models read execution.transactions by to_address:

SELECT max(block_timestamp) FROM dbt.⟨model⟩;
SELECT max(block_timestamp), count() FROM execution.logs
WHERE address = '⟨addr_no_0x_prefix⟩' AND block_timestamp > ⟨watermark⟩;

Never raise --max-slices-per-stage on the cron path. The cap exists so a multi-month gap is routed to full_refresh.py instead of being ground through the nightly window.

6. The live loop¶

Restart the live-loop workload. tag:live models freeze while it is down and catch up on restart. One replica, Recreate — never scale it, even temporarily: two pods would run the same incremental models against the same tables concurrently. Do not restart it twice in quick succession expecting faster catch-up; each restart is another gap.

7. Suspend drift, both directions¶

A pause set directly on the cluster vanishes at the next Terraform apply. A Terraform pause needs an apply to lift. Whichever you did, say so — there is no reconcile loop. Do not use the stack's cutover switch to pause just the cron: it zeroes the live loop and the server too.

8. Resuming an interrupted multi-model rebuild¶

A consumer rebuild stopped part-way leaves no usable resume state — target/refresh_state/ is on the pod's ephemeral volume. Rebuild the list from each model's own watermark:

SELECT 'model_a' AS m, max(date) FROM dbt.model_a
UNION ALL SELECT 'model_b', max(date) FROM dbt.model_b;

Models already at the expected date are done; the rest need re-running in DAG order.