Monitoring¶

The Gnosis Analytics platform uses Prometheus for metrics collection, Promtail and Loki for log aggregation, and configured alerting rules for proactive issue detection. This page documents the observability stack, key metrics, and alerting configuration.

Observability Stack¶

flowchart LR
    subgraph Services
        API[cerebro-api]
        IDX[Indexers]
        CRW[Crawlers]
    end
    subgraph Metrics
        PROM[Prometheus]
    end
    subgraph Logging
        PROMTAIL[Promtail]
        LOKI[Loki]
    end
    subgraph Alerting
        AM[Alertmanager]
    end

    API -->|metrics| PROM
    IDX -->|metrics| PROM
    CRW -->|metrics| PROM
    API -->|stdout/stderr| PROMTAIL
    IDX -->|stdout/stderr| PROMTAIL
    CRW -->|stdout/stderr| PROMTAIL
    PROMTAIL --> LOKI
    PROM --> AM
    AM -->|notifications| SLACK[Slack / Email]

Metrics Collection¶

Prometheus¶

Prometheus runs in the EKS cluster and scrapes metrics from all services via Kubernetes service discovery. It automatically discovers pods with the appropriate annotations.

Pod annotations for scraping:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
    prometheus.io/path: "/metrics"

Key Metrics¶

API Metrics¶

ClickHouse Query Metrics¶

Indexer Metrics¶

Crawler Metrics¶

Log Aggregation¶

Promtail + Loki¶

Promtail runs as a DaemonSet on every node, tailing container logs and shipping them to Loki for centralized storage and querying.

Log pipeline:

1. Containers write to stdout/stderr (standard Kubernetes logging)
2. Promtail discovers containers via Kubernetes API and tails their log files
3. Logs are enriched with labels: namespace, pod name, container name
4. Logs are pushed to Loki for storage
5. Logs are queryable via LogQL

Structured Logging¶

Services use structured JSON logging for machine-parseable log entries:

{
  "timestamp": "2024-03-15T10:30:45Z",
  "level": "INFO",
  "message": "Manifest refreshed successfully",
  "service": "cerebro-api",
  "models_loaded": 412,
  "duration_ms": 1250
}

Log Queries (LogQL)¶

Common LogQL queries for troubleshooting:

{namespace="cerebro", container="cerebro-api"} |= "ERROR"

{namespace="cerebro"} |= "ClickHouse" |= "error"

{container="cerebro-api"} |= "Manifest"

{namespace="indexers"} |= "block" | json | line_format "{{.block_number}}"

Alerting¶

Alertmanager¶

Prometheus Alertmanager handles alert routing and notification delivery. Alerts are routed to Slack channels and email based on severity.

Alert Rules¶

Critical Alerts¶

These alerts indicate service outages requiring immediate attention:

# Example Prometheus alert rule
groups:
  - name: cerebro-api
    rules:
      - alert: APIHighErrorRate
        expr: |
          sum(rate(http_requests_total{namespace="cerebro",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{namespace="cerebro"}[5m]))
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 5%"
          description: >
            The cerebro-api 5xx error rate has exceeded 5% for the last 5 minutes.
            Current rate: {{ $value | humanizePercentage }}.

Warning Alerts¶

These alerts indicate degraded performance or potential issues:

Informational Alerts¶

Notification Channels¶

Health Checks¶

API Health Check¶

The cerebro-api exposes a health check at the root endpoint:

curl https://api.analytics.gnosis.io/

{
  "status": "online",
  "service": "Gnosis Cerebro Data API",
  "docs": "/docs"
}

This endpoint is used by:

• Kubernetes readiness probe -- Determines if the pod should receive traffic
• Kubernetes liveness probe -- Determines if the pod should be restarted
• ALB health check -- Determines if the target is healthy

Container Health Checks¶

Each Docker container includes a built-in HEALTHCHECK instruction:

HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/ || exit 1

Kubernetes Probes¶

readinessProbe:
  httpGet:
    path: /
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 30
  failureThreshold: 5

Dashboards¶

Key monitoring dashboards:

Next Steps¶

• Troubleshooting -- Use monitoring data to diagnose issues
• Infrastructure -- Underlying platform architecture
• Deployment -- Deployment procedures and configuration