Running Models¶

Five scripts and the plain dbt run drive dbt-cerebro models in production. Which lever is safe depends on the model's materialization and on why you are running it. The decision tree lives on dbt reprocess after upstream repair; this page is the reference for each lever.

If you haven't read Incremental Strategies yet, do so first — it explains why the same model behaves differently across these runners.

TL;DR¶

Flag gotchas that have cost time: only dbt_incremental_runner.py accepts --project-dir / --profiles-dir — refresh.py and gap_window_refresh.py reject them and exit 2 at argparse. context.py takes no --project-dir. classify_failed_nodes.py takes --stash-dir, not --dir.

Mode 1 — Daily cron (run_dbt_observability.sh)¶

The daily CronJob at 06:00 UTC runs cron_preview.sh → scripts/run_dbt_observability.sh, which delegates each batch to scripts/refresh/dbt_incremental_runner.py. One attempt, no Kubernetes retries — the orchestrator owns retries internally (transients are retried once at --threads 1). For models with no microbatch annotation, the runner falls back to plain dbt run --select <batch> with no vars set — the daily incremental mode (delete+insert over the macro's lookback).

You don't normally invoke this manually. To reproduce a single batch the cron would run:

dbt run --select int_execution_blocks_daily

What ClickHouse sees:

INSERT INTO dbt.int_execution_blocks_daily SELECT ...
WHERE toStartOfMonth(toDate(block_timestamp)) >= ( -- last month's start
  SELECT toStartOfMonth(addDays(max(toDate(x1.date)), 0)) FROM dbt.int_execution_blocks_daily AS x1
)
AND toDate(block_timestamp) >= (
  SELECT addDays(max(toDate(x2.date)), 0) FROM dbt.int_execution_blocks_daily AS x2
);
ALTER TABLE dbt.int_execution_blocks_daily DELETE WHERE date IN (...inserted dates...);

The mutation deletes the previous version of the rewritten dates so RMT doesn't have to dedupe. Cheap on small daily windows; expensive on large windows (which is why the next modes exist).

When the run fails, The dbt daily run failed is the procedure: reading the log, classifying transient vs permanent, rerunning only what failed.

The live loop¶

Separately, a one-replica Deployment runs dbt run --select tag:live (19 models) every 45 seconds. It is Recreate and must never be scaled — two pods would run the same incremental models against the same tables. Restarting it opens a gap that closes on the next tick; restarting it twice in quick succession is two gaps. Anything windowed you run by hand should carry --exclude 'tag:live+'.

Mode 2 — Microbatch runner (dbt_incremental_runner.py)¶

For models tagged microbatch (and configured in meta.full_refresh.incremental in their schema.yml), the runner slices the gap between max(target_date) and today into per-day windows.

python /app/scripts/refresh/dbt_incremental_runner.py \
  --select int_consensus_validators_income_daily

Internally the runner emits one dbt run per slice, with incremental_end_date set:

dbt run --select int_consensus_validators_income_daily \
  --vars '{"incremental_end_date": "2026-04-21", "validator_index_start": 0, "validator_index_end": 100000}'
dbt run --select int_consensus_validators_income_daily \
  --vars '{"incremental_end_date": "2026-04-22", "validator_index_start": 0, "validator_index_end": 100000}'
...

Two important effects:

1. Strategy flips to append because the strategy expression is ('append' if (start_month or incremental_end_date) else 'delete+insert'). No mutation per slice.
2. apply_monthly_incremental_filter takes its no-overlap branch — WHERE date > max(target_date) AND date <= incremental_end_date. Re-running the same slice writes nothing (idempotent).

Common runner flags¶

When the runner refuses¶

gap is 615 day(s); exceeds --max-slices-per-stage=30

A refusal has three possible causes — no consumers, stalled, or dormant — and each has a different answer. The triage table is on The dbt daily run failed.

Mode 3 — Full-refresh batched (refresh.py)¶

For historical backfill into an empty or dropped table, monthly batches are the right granularity. Each model declares its history under meta.full_refresh in schema.yml:

- name: int_consensus_validators_income_daily
  meta:
    full_refresh:
      start_date: "2021-12-01"
      batch_months: 1
      stages:
        - name: validators_0_100k
          start_date: "2021-12-01"
          vars: { validator_index_start: 0, validator_index_end: 100000 }
        - name: validators_100k_200k
          ...

Run the whole annotated history:

python /app/scripts/full_refresh/refresh.py --select int_consensus_validators_income_daily

The runner iterates (stage × month) and emits one dbt run per batch with start_month / end_month and the stage's vars. Strategy flips to append.

Resume on failure¶

Each completed batch is written to target/refresh_state/. Re-invoking the same command in the same container picks up where it left off. In a one-shot job that state lives on the pod's ephemeral volume, so a new Job starts from scratch — rebuild the remaining list from each model's own max(date) instead. A pending run must be --resumed or --clear-stated, never deleted.

Transient retries¶

Code: 241 / 159 / 209 / 210 and MEMORY_LIMIT_EXCEEDED / OvercommitTracker errors are auto-retried with exponential backoff (30s → 60s → 120s → 240s → 480s, max 5 attempts). Logs show [transient] retry n/5 in Ns between attempts.

Restricting to a subset¶

# A single month
--start-date 2024-04-01 --end-date 2024-04-30

# A single stage
--stages validators_0_100k

# A single (stage, month)
--stages validators_0_100k --start-date 2024-04-01 --end-date 2024-04-30

Mode 4 — Gap-window refresh (gap_window_refresh.py)¶

For the case Modes 1–3 cannot handle: an indexer backfilled rows below a decode model's watermark. Decode models are append with a literal block watermark and no lookback, so nothing that runs forward will ever see those rows. This script drops the gap-month partition (which lowers the watermark) and re-runs scoped.

python /app/scripts/refresh/gap_window_refresh.py \
  --select ⟨decode_model⟩+ \
  --exclude 'tag:live+' \
  --months 2026-07-01,2026-08-01,2026-09-01 \
  --state /app/target/gap_refresh_state.json \
  --full-refresh-row-cap 0 \
  --dry-run

Every flag is load-bearing:

• --dry-run first, and read the ACTION column. The script branches three ways. Month-partitioned models get drop + scoped re-run; any append or delete+insert model with no start_month branch gets dbt run --full-refresh — a full drop and rebuild, guarded only by a row cap that defaults to 5,000,000.
• --full-refresh-row-cap 0 makes it skip and report those models instead of rebuilding them (--allow-big-full-refresh is the explicit override).
• --state is mandatory in-cluster: the default path is on the read-only image layer and the first save raises OSError — after partitions have already been dropped.
• --exclude 'tag:live+' is the correct selector; '*_live+' matches the wrong models.
• --months must span the gap month through the current month whenever a month boundary has been crossed, because cumulative models (the 39 that read {{ this }}) are wrong from the gap day forward.

Never run it on a table-materialized model. The full branch logic, hazards and verification queries are on dbt reprocess after upstream repair.

Mode 5 — Refill recovery (refill_after_price_gap.sh)¶

For a Dune prices skip specifically. Two-phase by design:

/app/scripts/maintenance/refill_after_price_gap.sh --from-date 2026-04-17

See Recovering from a Prices Gap for the full walk-through.

Plain dbt run recipes¶

For ad-hoc work outside the runners. Run context.py first; it tells you the model's materialization and the safe lever.

A single model with default daily behavior¶

dbt run --select int_execution_blocks_daily

A model and everything downstream of it¶

dbt run --select int_execution_token_prices_daily+

Full-refresh of a single incremental model (drops & rebuilds)¶

dbt run --select int_execution_blocks_daily --full-refresh

Compile only (debug the rendered SQL without writing)¶

dbt compile --select int_execution_tokens_balances_daily \
  --vars '{"price_lookback_days": 12}'

# Inspect target/compiled/.../<model>.sql

Append-mode rewrite of a single month (the Phase-1 primitive)¶

Always pass both start_month and end_month — start_month alone selects append plus the legacy lookback window and duplicates rows.

dbt run --select int_execution_tokens_balances_daily \
  --vars '{"start_month": "2026-04-01", "end_month": "2026-04-01"}'

dbt run-operation optimize_partition_final \
  --args '{database: dbt, table_name: int_execution_tokens_balances_daily, partition: "2026-04-01"}'

The OPTIMIZE collapses any duplicates RMT would otherwise merge lazily. Run it through the macro so it is synchronous; skip it only if you don't need immediate convergence.