ip-crawler¶

ip-crawler is a Python service that enriches peer IP addresses discovered by nebula with geolocation and network data from the ipinfo.io API, into crawlers_data.ipinfo. It feeds three public-dashboard HOPR marts.

Purpose¶

While nebula discovers which peers exist on the network and captures their IP addresses, ip-crawler adds geographic and organizational context to each IP. This enables analytics such as:

• Geographic distribution of Gnosis Chain nodes by country and city
• Hosting provider and ASN concentration analysis
• Network diversity metrics
• Identification of centralization risks

How It Works¶

1. Find un-enriched IPs — a server-side anti-join of nebula.visits against crawlers_data.ipinfo, bounded by a memory fuse. There is no state table and no cursor: the anti-join is the state.
2. Look up once, ever — an IP is queried once, including failures, which are written as permanent negatives so they are never retried.
3. Batch lookup — IPs go to the ipinfo.io API in batches, under two per-run caps.
4. Write — enriched rows land in crawlers_data.ipinfo.

Data Schema¶

The crawlers_data.ipinfo table stores enrichment results:

company, carrier and abuse_* are legitimately empty — the current ipinfo plan returns only the nine core fields. The table is a plain MergeTree and does not dedupe: readers need LIMIT 1 BY ip. There are 71 known legacy duplicate rows.

Configuration¶

Required Settings¶

ClickHouse Connection¶

Processing Settings¶

Fork Digest Configuration¶

Fork digests are updated when the Gnosis Chain undergoes protocol upgrades.

Operating and recovering¶

How it runs¶

A daily CronJob at 02:00 UTC, concurrencyPolicy: Forbid, with a 36,000 s deadline. Runs are stateless and idempotent: every run re-derives its work list from the anti-join, so re-running is always safe. There is no continuous mode in production.

Health — one log line¶

There are zero alert rules and no cron-miss rule. Health is the single run_summary JSON line each run emits. Alert-worthy: no run_summary for 26 h; exit_code != 0; truncated=true on the recent window; clickhouse_errors > 0. phases.sweep.truncated alone is normal while a backlog drains.

The data check is the same as every other source: max(created_at) on crawlers_data.ipinfo should be within a day.

Backfill¶

Backfill a window explicitly (always single-phase):

python -m src.crawler --since ⟨YYYY-MM-DD⟩ --until ⟨YYYY-MM-DD⟩

Run it as a one-shot job cloned from the CronJob (how). Forbid cannot see a hand-created Job, so space manual runs yourself.

The hopr preset has an ordering bug. It reads dbt.int_hopr_nodes and writes a table dbt reads back, so it should run after the dbt cron. It fires at 02:00; dbt runs at 06:00. Not currently harmful for the default preset, which reads nebula.visits, but do not add dbt-reading presets to this schedule without moving it.

Then dbt¶

The three HOPR marts are rebuilt by the nightly run; nothing to do by hand unless the gap is older than the models' incremental window — see dbt reprocess.

Redeploying¶

On an image roll nothing; the CronJob template changes and the 02:00 UTC run is the first on the new image. Apply outside 01:55-02:10 UTC and never while a Job is active. The decisive proof is the next run succeeds (morning check). Procedure: Redeploying a service.

ClickHouse Table Schemas¶