Coming soon. The Wiki is in active development, so features and these docs may still change. Early adopters: contact us for access.

Monitoring with Prometheus

Expose each server's metrics to Prometheus: live editing, background work on pages, backups, rate limits, caches and the server process

Each Wiki server can expose its metrics in the Prometheus format at /metrics. It's off by default, works on every license tier and in any license state, and isn't served in setup or recovery mode. Each server reports its own numbers, so scrape every server.

1. Turn it on

Choose one, or both:

VariableWhat
METRICS_ADDRA separate listener, such as :9090 or 10.0.0.5:9090, serving /metrics without authentication. It must differ from PORT. Bind it to a private interface or firewall it.
METRICS_TOKENAlso serves /metrics on the server's main port to requests with Authorization: Bearer <token>. At least 16 printable characters without spaces, for example openssl rand -hex 32.
# prometheus.yml
scrape_configs:
  - job_name: wiki
    static_configs:
      - targets: ["wiki-1.internal:9090", "wiki-2.internal:9090"]

  # or, on the main port with a token
  - job_name: wiki-token
    authorization: { credentials_file: /etc/prometheus/wiki-token }
    static_configs:
      - targets: ["wiki-1.internal:8080"]

/metrics is never reachable through the web app's address: Prometheus scrapes the servers directly. A scrape isn't rate limited and isn't counted in the metrics.

2. Metrics

Every metric name starts with depotex_, the name of the platform the Wiki is built on.

MetricWhat
depotex_collab_connectionsLive editing connections this server serves now
depotex_collab_updates_total, depotex_collab_update_bytes_totalEdits that browsers sent and that were stored, and their size in bytes
depotex_collab_closed_total{code}Live editing connections the server closed, by close code
depotex_collab_presence_dropped_totalCursor and presence messages dropped because a queue was full
depotex_wiki_doc_compactions_total, depotex_wiki_doc_compaction_failures_total{reason}Page documents whose stored edits were merged, and merges that failed
depotex_wiki_draft_snapshots_total, depotex_wiki_draft_snapshot_failures_total{reason}Draft snapshots taken (the draft history of a page), and snapshots that failed
depotex_backup_runs_total{kind, outcome}, depotex_backup_verifications_total{outcome}Backups taken and checked, by result
depotex_backup_duration_seconds{kind}, depotex_backup_last_size_bytesHow long backups take, and the size of the last one
depotex_backup_last_success_timestamp_secondsWhen the last backup succeeded
depotex_backup_cleanup_deleted_totalBackups the cleanup job deleted
depotex_backup_restore_runs_total{outcome}, depotex_backup_restore_duration_secondsRestores and how long they took
depotex_backup_test_restore_runs_total{outcome}, depotex_backup_test_restore_last_success_timestamp_secondsTest restores, and when the last one passed
depotex_ratelimit_requests_total{class, result}Requests checked against a budget, per kind of traffic and result: allowed, limited (refused) or dry_run (over, but let through by a dry run). A request counts once per budget it is checked against
depotex_ratelimit_fallback_total{reason}Checks a server made in its own memory although Redis is configured: redis_error, backoff (Redis skipped for a few seconds after an error) or canceled
depotex_ratelimit_redis_up1 while the server reaches Redis for rate limits, 0 while it counts on its own. Missing on servers without Redis
depotex_ratelimit_check_duration_seconds{store}How long a rate limit check took, in Redis or in memory
depotex_read_cache_hits_total{cache}, depotex_read_cache_misses_total{cache}Reads of settings answered from this server's memory, and reads that went to the database
depotex_read_cache_checks_total{result}, depotex_read_cache_notices_totalHow often the server checked whether its cached settings changed, and change notices it received
depotex_oauth_workload_exchanges_total{grant, endpoint, outcome}CI identity tokens exchanged for access tokens (see OAuth applications)
go_*, process_*The server process: memory, goroutines, CPU, open files

Metrics never carry who made a request, or which page or space it was about (no user, token, page or address labels). To see who was limited, use the Activity tab of Rate limits.

3. Alerting examples

# No successful backup for more than a day (with the scheduled backup on)
- alert: WikiBackupStale
  expr: time() - depotex_backup_last_success_timestamp_seconds > 26 * 3600
  for: 15m

# Pages' drafts can't be snapshotted
- alert: WikiDraftSnapshotsFailing
  expr: sum(rate(depotex_wiki_draft_snapshot_failures_total[15m])) > 0
  for: 30m

# A server lost Redis: rate limits are per server until it's back
- alert: WikiRateLimitRedisDown
  expr: depotex_ratelimit_redis_up == 0
  for: 5m