cryo-indexer¶

The cryo-indexer is the primary execution layer indexer for the Gnosis Analytics pipeline. It extracts blockchain data from an RPC node using the Cryo binary (a high-performance Rust data extraction tool) and loads it into ClickHouse.

Purpose¶

cryo-indexer handles the full lifecycle of execution layer data acquisition:

• Real-time indexing of new blocks as they are produced
• Historical bulk loading of past block ranges
• Automatic recovery from failures and incomplete ranges
• Validation of data integrity

Architecture¶

Blockchain RPC --> Cryo Binary --> Parquet Files --> ClickHouse
                                        |
                                  Worker Process
                                        |
                                  State Manager
                                        |
                                indexing_state table

The indexer orchestrates the Cryo binary to extract data from the RPC node into intermediate Parquet files, which are then loaded into ClickHouse. A single indexing_state table in ClickHouse serves as the source of truth for all processing state.

Key Design Decisions¶

• Blocks first -- Block headers are always processed before other datasets, because downstream datasets require valid block timestamps.
• 1000-block chunks -- All processing uses fixed 1000-block ranges for predictable resource usage.
• Atomic ranges -- A range is either fully completed or marked as failed. No partial writes.
• Single state table -- The indexing_state table tracks all datasets and ranges in one place.

Datasets¶

cryo-indexer supports 11 datasets organized into four indexing modes:

Indexing Modes¶

Dataset Reference¶

Operation Modes¶

Continuous (Default)¶

Real-time blockchain following for production deployments.

• Polls the chain tip every 10 seconds (configurable via POLL_INTERVAL)
• Waits for 12 block confirmations before indexing (configurable via CONFIRMATION_BLOCKS) to avoid reorg issues
• Processes in small batches (default 100 blocks) for low latency
• Single-threaded for stability
• Automatically resumes from the last indexed block on restart
• Resets stale processing jobs on startup

make continuous
# Or with custom settings:
make continuous MODE=full START_BLOCK=18000000

Historical¶

Fast bulk indexing for initial sync or backfilling specific block ranges.

• Supports parallel processing with multiple workers
• Automatically divides work into 1000-block chunks
• Built-in progress tracking with ETA calculations
• Strict timestamp validation at each step

make historical START_BLOCK=1000000 END_BLOCK=2000000 WORKERS=8

Maintain¶

Processes failed and pending ranges from the state table.

• Scans indexing_state for ranges marked as failed or pending
• Re-attempts each range with proper error handling
• Reports what was fixed and any remaining issues

make maintain
# Or for a specific range:
make maintain START_BLOCK=1000000 END_BLOCK=2000000 WORKERS=4

Validate (Read-Only)¶

Checks indexing progress and data integrity without modifying data.

make status

Configuration¶

Required Settings¶

Core Settings¶

Operation Settings¶

Performance Settings¶

State Management¶

All indexing state is tracked in the indexing_state table:

-- Composite key: (mode, dataset, start_block, end_block)
-- Status flow: pending -> processing -> completed | failed

On startup, all ranges stuck in processing state are automatically reset to pending, enabling self-healing after crashes.

Docker Deployment¶

# Build
docker-compose build

# Run migrations
docker-compose --profile migrations up migrations

# Start continuous indexing (minimal mode)
docker-compose up cryo-indexer-minimal

# Historical backfill
OPERATION=historical START_BLOCK=18000000 END_BLOCK=18100000 \
  docker-compose --profile historical up historical-job

Typical Deployment Workflow¶

1. Initial setup -- Build the image and run database migrations
2. Historical sync -- Bulk-load the desired block range with multiple workers
3. Maintenance pass -- Run maintain to retry any failed ranges
4. Switch to continuous -- Start following the chain tip in real time
5. Periodic maintenance -- Occasionally run maintain to clear any accumulated failures

ClickHouse Table Schemas¶

All tables are stored in the execution database.