cryo-indexer¶

The cryo-indexer is the primary execution layer indexer for the Gnosis Analytics pipeline. It extracts blockchain data from an RPC node using the Cryo binary (a high-performance Rust data extraction tool) and loads it into ClickHouse.

Purpose¶

cryo-indexer handles the full lifecycle of execution layer data acquisition:

• Real-time indexing of new blocks as they are produced
• Historical bulk loading of past block ranges
• Automatic recovery from failures and incomplete ranges
• Validation of data integrity

Architecture¶

Blockchain RPC --> Cryo Binary --> Parquet Files --> ClickHouse
                                        |
                                  Worker Process
                                        |
                                  State Manager
                                        |
                                indexing_state table

The indexer orchestrates the Cryo binary to extract data from the RPC node into intermediate Parquet files, which are then loaded into ClickHouse. A single indexing_state table in ClickHouse serves as the source of truth for all processing state.

Key Design Decisions¶

• Blocks first -- Block headers are always processed before other datasets, because downstream datasets require valid block timestamps.
• 1000-block chunks -- All processing uses fixed 1000-block ranges for predictable resource usage.
• Atomic ranges -- A range is either fully completed or marked as failed. No partial writes.
• Single state table -- The indexing_state table tracks all datasets and ranges in one place.

Datasets¶

cryo-indexer supports 11 datasets organized into four indexing modes:

Indexing Modes¶

Dataset Reference¶

Operating and recovering¶

How it runs¶

One image and no command: the entrypoint dispatches on OPERATION ∈ continuous | historical | maintain | auto-maintain | validate. DATASETS is honoured only when MODE=custom; MODE=minimal silently ignores it and gives you blocks, transactions, logs. All state is <db>.indexing_state; the pod holds nothing.

Three continuous writers, one per database: Gnosis (execution, CONFIRMATION_BLOCKS=720, MODE=custom with six datasets — blocks, transactions, logs, contracts, native_transfers, traces), Gnosis live (execution_live, CONFIRMATION_BLOCKS=6, two-day TTL) and Celo (celo_execution, MODE=minimal). Each chain also has an auto-maintain CronJob, seven times a day, that re-extracts recent failed or missing ranges. Cryo itself and its two patches live in cryo-base; odd Cryo behaviour starts there.

Single writer per database, Recreate, one replica — enforced by convention only. The target tables do not dedupe re-inserted rows, and maintain DELETEs a range before it claims it.

Health¶

Config failures are narrow: the entrypoint exits 1 if ETH_RPC_URL or CLICKHOUSE_HOST is empty. A failed secret sync shows as a container-config error, not a crash loop.

The authoritative liveness check needs only warehouse access:

SELECT dataset, max(end_block) AS highest_completed, max(created_at) AS last_write
FROM execution.indexing_state WHERE status = 'completed' GROUP BY dataset ORDER BY dataset;

last_write older than a few minutes means the writer is gone (POLL_INTERVAL=60, BATCH_SIZE=100). Repeat for execution_live and celo_execution. Head lag is ~66 minutes on execution (720 confirmations at 5 s) and ~1 minute on execution_live; neither is a stall.

A pod that is alive but wedged: restart the workload. Never above one replica and never a rolling update.

Detecting a gap¶

Per-dataset batch coverage, run once per dataset:

WITH expected AS (SELECT toUInt32(⟨first_block⟩ + number * 100) AS s FROM numbers(⟨n_batches⟩))
SELECT s AS missing_start_block FROM expected
WHERE s NOT IN (SELECT DISTINCT start_block FROM execution.indexing_state WHERE dataset = 'transactions')
ORDER BY s;

Repeat for blocks, logs, and on execution also contracts, native_transfers, traces. Derive the bounds from the chain; do not copy numbers. A hole in one dataset does not imply a hole in the others.

Is a transaction-less batch a real hole or a quiet chain? Gas burned means transactions existed:

SELECT intDiv(block_number,100)*100 AS batch, count() AS blocks, sum(gas_used) AS gas
FROM execution.blocks WHERE block_number >= ⟨lo⟩ AND block_number < ⟨hi⟩
GROUP BY batch HAVING gas > 0 ORDER BY batch;

OPERATION=validate with a block range writes nothing and exits non-zero when gaps exist, so it can run inside the live pod. Also read the continuous log for max retries, All datasets exhausted, Moving to next range — the loop gives up on a dataset after MAX_RETRIES and moves the pointer forward, leaving a hole behind.

Repairing¶

Recent gaps: auto-maintain. Its reach is AUTO_MAINTAIN_LOOKBACK_HOURS × 720 blocks, and 720 blocks/hour is hardcoded for Gnosis' 5 s blocks — so 48 h is real on Gnosis and only ~9.6 h on Celo. Anything older is permanently out of its range. Trigger an extra run by cloning its CronJob (one-shot jobs).

Older gaps: a scoped maintain. It selects every non-completed range (processing included) and DELETEs it before re-extracting, with no claim, so the live writer for that database must be stopped first — the stop sequence and the one-shot recipe are on the one-shot jobs page. Chunk to a few hundred thousand blocks per job.

Gnosis needs MODE=custom with all six live datasets — MODE=minimal repairs three of six and leaves the rest holed. Celo runs MODE=minimal live, so there minimal is the whole set; its RPC settings are 20× more aggressive than Gnosis' — keep them.

Corrupt range¶

SELECT dataset, start_block, end_block, argMax(status, created_at) AS st
FROM execution.indexing_state
WHERE start_block >= ⟨lo⟩ AND end_block <= ⟨hi⟩
GROUP BY dataset, start_block, end_block ORDER BY dataset, start_block;

INSERT INTO execution.indexing_state
  (dataset, start_block, end_block, status, error_message, attempt_count)
VALUES
  ('blocks',       ⟨lo⟩, ⟨hi⟩, 'failed', 'manual re-open: bad RPC data', 1),
  ('transactions', ⟨lo⟩, ⟨hi⟩, 'failed', 'manual re-open: bad RPC data', 1),
  ('logs',         ⟨lo⟩, ⟨hi⟩, 'failed', 'manual re-open: bad RPC data', 1);

Then the scoped maintain above.

Order matters: blocks must be correct before transactions / logs. The other datasets derive block_timestamp by joining blocks and hard-fail if any block has an invalid timestamp. That failure logs as CRITICAL: Cannot add timestamps for <table>! ... Process blocks first! on transactions, which points diagnosis at the wrong dataset. Garbage timestamps, with the indexer's own predicate:

SELECT count() FROM execution.blocks FINAL
WHERE block_number >= ⟨lo⟩ AND block_number < ⟨hi⟩
  AND (timestamp IS NULL OR timestamp = 0 OR toDateTime(timestamp) <= toDateTime('1971-01-01'));

Cold partitions need image 19e81f5 or later. Before that, the delete removed the rows but not the Keeper dedup block-id, so the re-insert was silently refused while the client saw written_rows=N. The only trace is system.part_log.error = 389.

Verify:

SELECT count() raw, countDistinct(block_number) d, max(block_number)-min(block_number)+1 span
FROM execution.blocks WHERE block_number >= ⟨lo⟩ AND block_number < ⟨hi⟩;   -- want raw = d = span

SELECT status, rows_indexed FROM execution.indexing_state
WHERE dataset='transactions' AND start_block=⟨lo⟩ AND end_block=⟨hi⟩
ORDER BY created_at DESC LIMIT 1;   -- want 'completed' with rows_indexed > 0

A completed with rows_indexed = 0 is the silent-failure shape auto-maintain hunts.

Restart¶

Killing mid-range is safe for correctness, but SIGTERM exits immediately without finishing or failing the in-flight range — it stays processing until STUCK_RANGE_TIMEOUT_HOURS (2). Until then it is invisible to gap detection, and find_gaps deliberately clamps below anything still processing so a repair cannot stomp a live range. If you cannot wait two hours before a scoped repair over that range, re-open it as failed with the INSERT above.

A durable stop or start is a replica change in the stack, applied (Deployment); a live scale is [drift]. Restart resumes from the minimum across datasets of max(end_block) WHERE status='completed', aligned up to a batch boundary. On celo_execution, START_BLOCK is a floor, not a resume point.

Then dbt¶

A raw repair does not fix the decode layer — decode models are append with an embedded watermark and cannot see anything backfilled below it. Go to dbt reprocess. Do not run the daily microbatch runner: it only advances watermarks and produces a completely green run that fixes nothing.

Redeploying¶

On an image roll the continuous writers restart (the auto-maintain CronJob picks the image up at its next slot). Apply outside the two hours before an auto-maintain slot, in an idle gap between ranges. The decisive proof is grid coverage over the restart window whole and no orphan range. Procedure: Redeploying a service.

Configuration¶

Required Settings¶

Core Settings¶

Operation Settings¶

Performance Settings¶

The production deployment overrides several code defaults; the values that explain healthy numbers are listed.

State Management¶

All indexing state is tracked in the indexing_state table:

-- Composite key: (mode, dataset, start_block, end_block)
-- Status flow: pending -> processing -> completed | failed

The table is append-only: the current status of a range is its latest row (argMax(status, created_at)). A range left in processing by an unclean stop becomes eligible for repair only after STUCK_RANGE_TIMEOUT_HOURS; it is not reset on startup.

ClickHouse Table Schemas¶

All tables are stored in the execution database.