ip-crawler¶

ip-crawler is a Python service that enriches peer IP addresses discovered by nebula with geolocation and network data from the ipinfo.io API.

Purpose¶

While nebula discovers which peers exist on the network and captures their IP addresses, ip-crawler adds geographic and organizational context to each IP. This enables analytics such as:

• Geographic distribution of Gnosis Chain nodes by country and city
• Hosting provider and ASN concentration analysis
• Network diversity metrics
• Identification of centralization risks

How It Works¶

1. Query for new IPs -- ip-crawler reads the nebula.visits table and identifies IP addresses that have not yet been enriched
2. Filter by fork digest -- Only IPs associated with configured Gnosis Chain fork digests are processed
3. Batch lookup -- IPs are sent to the ipinfo.io API in batches (default 50 per batch)
4. Rate limiting -- Requests are throttled to 10 per second to respect API limits
5. Store results -- Enriched data is written to the crawlers_data.ipinfo table
6. Incremental processing -- The table is processed month-by-month to avoid memory issues with large datasets

Operation Modes¶

Continuous Mode (Default)¶

Runs as a persistent daemon, processing new batches every 60 seconds:

docker-compose up -d ip-crawler

The crawler automatically:

• Detects new IPs from nebula.visits
• Processes them in batches
• Sleeps between cycles
• Resumes from the last processed partition on restart

One-Time Mode¶

Processes a single batch of IPs and exits. Useful for scheduled jobs (cron, Kubernetes CronJob) or testing:

# CLI
python -m src.crawler --once --batch-size 200

# Docker
docker-compose run --rm -e CRAWLER_MODE=once ip-crawler

One-time mode produces a summary report and saves statistics to logs/last_run_stats.json.

Data Schema¶

The crawlers_data.ipinfo table stores enrichment results:

Configuration¶

Required Settings¶

ClickHouse Connection¶

Processing Settings¶

Fork Digest Configuration¶

Fork digests are updated when the Gnosis Chain undergoes protocol upgrades. The variable can be changed at runtime -- the crawler detects the update automatically.

Incremental Processing¶

To handle the potentially large nebula.visits table without exceeding memory limits, ip-crawler processes data month-by-month:

1. Queries are scoped to a single month partition at a time
2. Processing state is saved to logs/partition_state.json
3. The crawler automatically resumes from the last processed month on restart
4. Once a month is fully processed, it moves to the next

Monitoring¶

Logs¶

# View container logs
docker-compose logs -f ip-crawler

Log files are stored in the logs/ directory:

Health Checks¶

The container includes a health check that verifies the crawler process is running by checking for the existence of the health log file.

Docker Deployment¶

# Setup
cp .env.example .env
# Edit .env with ClickHouse credentials and ipinfo.io token

# Run in continuous mode
docker-compose up -d

# Run one-time batch
docker-compose run --rm -e CRAWLER_MODE=once ip-crawler

ClickHouse Table Schemas¶