Each Depotex server can expose its metrics in the Prometheus format at /metrics. It's off by default, works on every license tier and in any license state, and isn't served in setup or recovery mode. Each server reports its own numbers, so scrape every server.
1. Turn it on
Choose one, or both:
| Variable | What |
|---|---|
| METRICS_ADDR | A separate listener, such as :9090 or 10.0.0.5:9090, serving /metrics without authentication. It must differ from PORT. Bind it to a private interface or firewall it. |
| METRICS_TOKEN | Also serves /metrics on the server's main port to requests with Authorization: Bearer <token>. At least 16 characters, for example openssl rand -hex 32. |
# prometheus.yml
scrape_configs:
- job_name: depotex
static_configs:
- targets: ["depotex-1.internal:9090", "depotex-2.internal:9090"]
# or, on the main port with a token
- job_name: depotex-token
authorization: { credentials_file: /etc/prometheus/depotex-token }
static_configs:
- targets: ["depotex-1.internal:8080"]/metrics is never reachable through the web app's address or the ingress: Prometheus scrapes the servers directly. A scrape isn't rate limited and isn't counted in the metrics.
On Kubernetes, set backend.metrics.enabled: true in the Helm chart: see Docker & Kubernetes.
2. Metrics
| Metric | What |
|---|---|
| depotex_ratelimit_requests_total{class, result} | Requests checked against a budget, per kind of traffic and result: allowed, limited (refused) or dry_run (over, but let through by a dry run). A request counts once per budget it is checked against |
| depotex_ratelimit_fallback_total{reason} | Checks a server made in its own memory although Redis is configured: redis_error, backoff (Redis skipped for a few seconds after an error) or canceled |
| depotex_ratelimit_redis_up | 1 while the server reaches Redis for rate limits, 0 while it counts on its own. Missing on servers without Redis |
| depotex_ratelimit_check_duration_seconds{store} | How long a rate limit check took, in Redis or in memory |
| go_*, process_* | The server process: memory, goroutines, CPU, open files |
Metrics never carry who made a request (no user, token or address labels). To see who was limited, use the Activity tab of Rate limits.
3. Alerting examples
# A server lost Redis: limits are per server until it's back
- alert: DepotexRateLimitRedisDown
expr: depotex_ratelimit_redis_up == 0
for: 5m
# Many requests refused
- alert: DepotexRateLimited
expr: sum by (class) (rate(depotex_ratelimit_requests_total{result="limited"}[5m])) > 10
for: 10m