Crawlers Module¶

The Crawlers module contains approximately 9 models that aggregate and enrich external reference data from third-party sources. Its primary dataset is the Dune Analytics label directory with over 5.3 million address labels, supplemented by price feeds, token supply data, and other reference datasets used to enrich on-chain analytics across the platform.

Data Sources¶

All data is sourced from the crawlers_data ClickHouse database, which is populated by scheduled crawlers that fetch and sync external datasets:

Model Categories¶

Country

Cow

Data

Dune

Ember

Ipinfo

Probelab

Using Labels via MCP¶

The Cerebro MCP server exposes the Dune label dataset through the resolve_address tool:

-- Look up a specific address
resolve_address("0x9c58bacc331c9aa871afd802db6379a98e80cedb")

-- Search by name
resolve_address("Uniswap")
resolve_address("Agave")

The get_token_metadata tool also leverages crawler price data:

get_token_metadata("GNO")
get_token_metadata("USDC")

Data Freshness¶

Crawler datasets are refreshed on different schedules:

Key Models Reference¶

Query Examples¶

Look up labels for a set of addresses:

SELECT address, name, category
FROM dbt.int_crawlers_dune_labels_deduplicated
WHERE address IN ('0x...', '0x...')

Get daily GNO price:

SELECT dt, avg_price_usd, close_price_usd
FROM dbt.int_crawlers_prices_daily
WHERE token_symbol = 'GNO'
  AND dt >= today() - 30
ORDER BY dt

Track GNO staking ratio over time:

SELECT dt, total_supply, staked_supply,
       round(staked_supply / total_supply * 100, 2) AS staking_ratio_pct
FROM dbt.int_crawlers_gno_supply_daily
WHERE dt >= today() - 90
ORDER BY dt

Search for protocol contracts:

SELECT address, name, category
FROM dbt.int_crawlers_dune_labels_deduplicated
WHERE category = 'dex'
  AND name ILIKE '%balancer%'

Related Modules¶

• Execution -- On-chain contract data enriched with Dune labels
• Bridges -- Bridge flow data originates from crawler feeds
• Contracts -- Contract classification uses label data for protocol identification
• ESG -- Ember electricity data is also ingested through the crawler pipeline