Node Population Estimation¶

The Hidden-Node Problem¶

Network crawlers such as Nebula traverse the Gnosis Chain's libp2p DHT and record every peer they can reach. However, a significant fraction of validators sit behind NAT gateways, firewalls, or restrictive cloud security groups and are never directly contacted. Reporting only the observed node count would systematically underestimate the true network size and, by extension, the energy footprint.

graph LR
    A[Nebula Crawler] -->|Discovers| B[Visible Nodes]
    A -.->|Cannot Reach| C[Hidden Nodes]
    B --> D[Observed Count S_obs]
    C --> E[Estimated via Chao-1]
    D --> F[Total Population Estimate]
    E --> F

Chao-1 Nonparametric Estimator¶

The module applies the Chao-1 estimator (Chao, 1984), a species-richness method from ecology that infers the number of unseen classes from the frequency distribution of rare observations. In our context, a "class" is a unique node and the observation frequency is the number of independent crawl sessions in which that node was successfully contacted.

Definitions¶

Standard Chao-1 Formula¶

The standard Chao-1 estimate of total node population is:

Bias-Corrected Form¶

When \(f_2 = 0\) (no doubletons), the bias-corrected form is used:

Variance and Confidence Intervals¶

The analytical variance of the Chao-1 estimator is:

A 95 % confidence interval is constructed using a log-transformation to ensure the lower bound never falls below S_obs:

where \(C = \exp\!\bigl(1.96 \sqrt{\ln(1 + \widehat{\text{Var}} / (\hat{S}_{\text{Chao1}} - S_{\text{obs}})^2)}\bigr)\).

Failure-Recovery Augmentation¶

Not every failed connection represents a permanently invisible node. Some failures are transient and can be resolved by retry logic. The recovery layer re-attempts failed peers and assigns a recovery rate based on the failure type:

Recovered Count Formula¶

The recovered node count for each failure category is:

The total adjusted population becomes:

dbt Implementation¶

The Chao-1 estimation is split across two dbt models: a staging model that consolidates raw crawl data and an intermediate model that computes the estimator.

stg_chao1_observers¶

This staging model consolidates crawl data from multiple independent observer instances into a single unified stream. Each observer contributes its own perspective of the network, which is essential for the frequency-based estimator.

-- stg_chao1_observers.sql
-- Consolidates crawl data from multiple observer instances

WITH observer_alpha AS (
    SELECT
        crawl_timestamp,
        peer_id,
        ip_address,
        'alpha' AS observer_id
    FROM {{ source('nebula', 'crawl_results_alpha') }}
),

observer_beta AS (
    SELECT
        crawl_timestamp,
        peer_id,
        ip_address,
        'beta' AS observer_id
    FROM {{ source('nebula', 'crawl_results_beta') }}
)

SELECT * FROM observer_alpha
UNION ALL
SELECT * FROM observer_beta

Output columns:

int_esg_node_population_chao1¶

This intermediate model groups crawl observations into hourly windows and applies the Chao-1 estimator within each window.

-- int_esg_node_population_chao1.sql
-- Applies Chao-1 estimator in hourly windows

WITH peer_frequencies AS (
    -- Count how many sessions each peer was seen in per hour
    SELECT
        toStartOfHour(crawl_timestamp) AS hour_timestamp,
        peer_id,
        count(DISTINCT observer_id) AS session_count
    FROM {{ ref('stg_chao1_observers') }}
    GROUP BY hour_timestamp, peer_id
),

singletons AS (
    -- F1: peers seen in exactly 1 session
    SELECT
        hour_timestamp,
        countIf(session_count = 1) AS f1
    FROM peer_frequencies
    GROUP BY hour_timestamp
),

doubletons AS (
    -- F2: peers seen in exactly 2 sessions
    SELECT
        hour_timestamp,
        countIf(session_count = 2) AS f2
    FROM peer_frequencies
    GROUP BY hour_timestamp
),

observed AS (
    -- S_obs: total distinct peers per hour
    SELECT
        hour_timestamp,
        count(DISTINCT peer_id) AS s_obs
    FROM peer_frequencies
    GROUP BY hour_timestamp
)

SELECT
    o.hour_timestamp,
    o.s_obs,
    s.f1,
    d.f2,
    -- Chao-1 estimate (bias-corrected when f2 = 0)
    CASE
        WHEN d.f2 > 0
            THEN o.s_obs + (pow(s.f1, 2) / (2.0 * d.f2))
        ELSE
            o.s_obs + (s.f1 * (s.f1 - 1)) / 2.0
    END AS chao1_estimate,
    -- 7-day moving average for smoothing
    avg(
        CASE
            WHEN d.f2 > 0
                THEN o.s_obs + (pow(s.f1, 2) / (2.0 * d.f2))
            ELSE
                o.s_obs + (s.f1 * (s.f1 - 1)) / 2.0
        END
    ) OVER (
        ORDER BY o.hour_timestamp
        ROWS BETWEEN 167 PRECEDING AND CURRENT ROW  -- 7 days * 24 hours
    ) AS chao1_estimate_7day_avg
FROM observed o
LEFT JOIN singletons s ON o.hour_timestamp = s.hour_timestamp
LEFT JOIN doubletons d ON o.hour_timestamp = d.hour_timestamp
ORDER BY o.hour_timestamp

Output columns:

Typical Results¶

The table below shows a representative breakdown for a recent estimation window:

pie title Node Population Breakdown
    "Directly Observed" : 1200
    "Hidden (Chao-1)" : 300
    "Recovered (Retries)" : 300