beacon-indexer¶

The beacon-indexer is a consensus layer indexer that fetches data from a beacon node REST API and stores it in ClickHouse or Parquet files. It follows an ELT (Extract, Load, Transform) pattern with fork-aware parsing and automatic network detection.

Purpose¶

beacon-indexer handles the acquisition of all consensus layer data:

• Validator sets, balances, and status changes
• Block proposals and attestations
• Sync committee participation
• Rewards and penalties
• Deposits, withdrawals, and voluntary exits

Architecture¶

The indexer operates in two distinct phases:

Phase 1: Load (Extract + Load)
    Beacon Node REST API --> Raw JSON --> ClickHouse raw_* tables

Phase 2: Transform
    raw_* tables --> Fork-aware parsing --> Structured tables

This ELT separation means raw data is preserved exactly as received from the beacon API. Transformation logic can be updated and re-run independently without re-fetching data from the beacon node.

Storage Backends¶

beacon-indexer supports two storage backends, selected via the STORAGE_BACKEND environment variable:

The ClickHouse backend provides fork-aware chunk-based processing with concurrent access and monitoring. The Parquet backend produces portable columnar files compatible with pandas, DuckDB, and Spark.

Fork Awareness¶

The beacon chain has undergone multiple protocol upgrades (forks), each introducing new data fields and structures. beacon-indexer automatically detects the fork version for each slot and applies the appropriate parser.

Supported Networks¶

Network parameters are auto-detected from the beacon chain genesis data. No manual configuration is required.

Operations¶

Migration¶

Creates the database schema (ClickHouse backend only):

make migration

Backfill¶

Loads historical raw data from the beacon API for a specified slot range:

make backfill

Transform¶

Parses raw data into structured tables with fork-appropriate field extraction:

make transform

Real-time¶

Continuously follows the chain head, loading new slots as they are produced:

make realtime

Maintenance¶

The maintenance system provides tools for data integrity checking and recovery:

# Check system health
make maintenance-status

# Find and fix failed chunks
make maintenance-fix

# Fix recent failures (last week)
make maintenance-fix-recent

# Analyze data gaps
make maintenance-gaps

Maintenance supports dry-run mode (--dry-run) to preview what would be changed without modifying data.

Configuration¶

Required Settings¶

ClickHouse Settings (when using ClickHouse backend)¶

Parquet Settings (when using Parquet backend)¶

Processing Settings¶

Data Flow¶

1. Load phase -- The backfill or realtime command fetches raw beacon API responses and stores them in raw_* tables (ClickHouse) or raw Parquet files.
2. Transform phase -- The transform command reads raw data, detects the fork version for each slot, and writes parsed data into structured tables (blocks, validators, attestations, etc.).
3. State tracking -- Chunk-based state tracking records which slot ranges have been loaded and transformed, enabling resumability.

Docker Deployment¶

# Build the image
make build

# Run the full pipeline
make migration       # Create schema (ClickHouse only)
make backfill        # Load historical data
make transform       # Parse raw data
make realtime        # Follow chain head

# Maintenance
make maintenance-status
make maintenance-fix

ClickHouse Table Schemas¶

All tables are stored in the consensus database.