beacon-indexer¶

The beacon-indexer is a consensus layer indexer that fetches data from a beacon node REST API and stores it in ClickHouse or Parquet files. It follows an ELT (Extract, Load, Transform) pattern with fork-aware parsing and automatic network detection.

Purpose¶

beacon-indexer handles the acquisition of all consensus layer data:

• Validator sets, balances, and status changes
• Block proposals and attestations
• Sync committee participation
• Rewards and penalties
• Deposits, withdrawals, and voluntary exits

Architecture¶

The indexer operates in two distinct phases:

Phase 1: Load (Extract + Load)
    Beacon Node REST API --> Raw JSON --> ClickHouse raw_* tables

Phase 2: Transform
    raw_* tables --> Fork-aware parsing --> Structured tables

This ELT separation means raw data is preserved exactly as received from the beacon API. Transformation logic can be updated and re-run independently without re-fetching data from the beacon node.

Storage Backends¶

beacon-indexer supports two storage backends, selected via the STORAGE_BACKEND environment variable:

The ClickHouse backend provides fork-aware chunk-based processing with concurrent access and monitoring. The Parquet backend produces portable columnar files compatible with pandas, DuckDB, and Spark.

Fork Awareness¶

The beacon chain has undergone multiple protocol upgrades (forks), each introducing new data fields and structures. beacon-indexer automatically detects the fork version for each slot and applies the appropriate parser.

Supported Networks¶

Network parameters are auto-detected from the beacon chain genesis data. No manual configuration is required.

Operating and recovering¶

How it runs¶

Three workloads, one image, all writing consensus:

State is two tables and nothing in the pod: consensus.load_state_chunks (per loader: pending | claimed | completed | failed) and consensus.transformer_progress. Always query both with FINAL — they keep every status transition, so without it you read history, not state.

The 02:00 Job takes ~4h11m; incomplete until ~06:15 is normal. Single writer per loader, Recreate, one replica.

Health¶

The Deployments carry no liveness or readiness probes by design, so a Running pod is not proof of progress. Ground truth is the raw frontier:

SELECT 'raw_blocks' AS t, max(slot) AS max_slot, max(slot_timestamp) AS latest FROM consensus.raw_blocks
UNION ALL SELECT 'raw_rewards', max(slot), max(slot_timestamp) FROM consensus.raw_rewards
UNION ALL SELECT 'raw_data_column_sidecars', max(slot), max(slot_timestamp) FROM consensus.raw_data_column_sidecars
UNION ALL SELECT 'raw_validators', max(slot), max(slot_timestamp) FROM consensus.raw_validators;

raw_blocks should advance ~100 slots every ~8–10 min and sit ~700–800 slots behind head (REALTIME_SLOT_DELAY=700, about 67 minutes at 5 s slots).

Detecting a gap¶

Every 100-slot chunk on the grid with no completed row, per loader:

SELECT loader, expected_start, expected_start + 99 AS expected_end
FROM (
  SELECT arrayJoin(['blocks','rewards','data_column_sidecars']) AS loader,
         ⟨LO⟩ + number * 100 AS expected_start
  FROM numbers(⟨window_width/100⟩)
) AS e
LEFT ANTI JOIN (
  SELECT loader_name AS loader, start_slot AS expected_start
  FROM consensus.load_state_chunks FINAL
  WHERE status = 'completed' AND start_slot >= ⟨LO⟩ AND start_slot < ⟨HI⟩
) AS c USING (loader, expected_start)
ORDER BY loader, expected_start;

⟨LO⟩ must be a multiple of 100. The last chunk of any backfill is truncated to end_slot - 1, so a single non-grid chunk at a previous backfill's boundary can show as a false positive — check it against load_state_chunks before repairing.

Chunks that exist but are not completed — the status tells you which repair applies:

SELECT loader_name, status, count() AS chunks, min(start_slot), max(end_slot)
FROM consensus.load_state_chunks FINAL
WHERE start_slot >= ⟨LO⟩ AND end_slot < ⟨HI⟩
GROUP BY loader_name, status ORDER BY loader_name, status;

Validators are one snapshot at the last slot of each UTC day, so a missing date is a gap:

SELECT toDate(slot_timestamp) AS d, count() AS snapshots
FROM consensus.raw_validators FINAL
WHERE slot_timestamp >= now() - INTERVAL 45 DAY GROUP BY d ORDER BY d;

maintain check --start-slot ⟨LO⟩ --end-slot ⟨HI⟩ --detailed finds failed and completed-but-untransformed chunks. It does not detect missing chunk rows — only the SQL above does.

Repairing¶

Run these as one-shot jobs cloned from the validators CronJob (how). Both slot arguments are required on every verb. --end-slot is exclusive, and chunk selection is start_slot >= LO AND end_slot <= HI, so a partially overlapping chunk is silently skipped. Keep both on 100-slot boundaries, strictly behind the live frontier.

Corrupt range¶

Raw tables are ReplacingMergeTree ORDER BY (slot, payload_hash), so two different payloads for one slot coexist — they are not deduped against each other. That is the bad-RPC signature:

SELECT slot, count() AS payload_variants FROM consensus.raw_blocks
WHERE slot BETWEEN ⟨LO⟩ AND ⟨HI⟩ GROUP BY slot HAVING payload_variants > 1 ORDER BY slot LIMIT 50;

Repair is maintain fix … --force --loaders blocks, always --dry-run first.

Verifying a repair: a duplicate scan right after a reload gives false positives, because the raw tables dedupe only at merge time. Judge by chunk status instead:

SELECT loader_name, status, count() FROM consensus.load_state_chunks FINAL
WHERE start_slot >= ⟨LO⟩ AND end_slot < ⟨HI⟩ AND loader_name = 'blocks'
GROUP BY loader_name, status;    -- want: all 'completed'

Re-deriving transformed tables without refetching¶

The real procedure is two statements; the third step is nothing, because the running transformer picks the chunks up on its next 10-second poll:

-- 1. delete the wrong rows from ONLY the affected transformed tables, one statement each
ALTER TABLE consensus.blocks DELETE WHERE slot BETWEEN ⟨LO⟩ AND ⟨HI⟩;

-- 2. un-complete the transform bookkeeping for exactly those chunks
INSERT INTO consensus.transformer_progress
  (raw_table_name, start_slot, end_slot, status, processed_count, failed_count, error_message, processed_at)
SELECT 'raw_blocks', start_slot, end_slot, 'failed', 0, 0, 'manual scoped reprocess', now()
FROM consensus.load_state_chunks FINAL
WHERE loader_name = 'blocks' AND status = 'completed'
  AND start_slot >= ⟨LO⟩ AND end_slot <= ⟨HI⟩;

TRANSFORM_CHUNKS_PER_BATCH is pinned to 1 — that is memory protection, not a tuning oversight, so a large range re-derives slowly. Each validators chunk parses a full ~400k-entry beacon state. Never TRUNCATE transformer_progress to force a re-derive; that is a full refresh of all history.

Transform lagging behind raw¶

SELECT
  (SELECT max(end_slot) FROM consensus.load_state_chunks FINAL WHERE loader_name='blocks' AND status='completed') AS raw_frontier,
  (SELECT max(end_slot) FROM consensus.transformer_progress FINAL WHERE raw_table_name='raw_blocks' AND status='completed') AS transform_frontier;

Swap the loader to find which one is lagging — the Prometheus alert takes max() across all table labels and hides exactly this.

OOMKilled on the transform workload is the classic stall. Restarting self-heals: on startup it resets any processing row older than 30 minutes back to failed, making those chunks eligible again. Check nobody raised TRANSFORM_CHUNKS_PER_BATCH above 1 before restarting.

Restart — the mandatory reset¶

An unclean kill leaves chunks in claimed, and nothing reaps them. maintain fix cannot see them (it selects failed). After any unclean kill:

# both slot arguments are REQUIRED
python -m src.main maintain reset --start-slot 0 --end-slot 999999999 --status claimed

Then dbt¶

consensus is a dbt source. Check each intermediate's strategy with context.py first, then follow dbt reprocess.

Redeploying¶

On an image roll realtime and transform restart. Apply outside 02:00-06:15 UTC, in the idle gap between chunks. The decisive proof is no chunk hole across the restart and the transform caught up. Procedure: Redeploying a service.

Configuration¶

Required Settings¶

ClickHouse Settings (when using ClickHouse backend)¶

Parquet Settings (when using Parquet backend)¶

Processing Settings¶

Data Flow¶

1. Load phase -- The backfill or realtime command fetches raw beacon API responses and stores them in raw_* tables (ClickHouse) or raw Parquet files.
2. Transform phase -- The transform command reads raw data, detects the fork version for each slot, and writes parsed data into structured tables (blocks, validators, attestations, etc.).
3. State tracking -- Chunk-based state tracking records which slot ranges have been loaded and transformed, enabling resumability.

ClickHouse Table Schemas¶

All tables are stored in the consensus database.