HLD · 20 min read

Design A Metrics Monitoring and Alerting System

System design for a scalable metrics monitoring and alerting System

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. What Makes Time-Series Data Different
  6. 3. Capacity Estimation
  7. 4. Push vs Pull Model
  8. Pull Model (Prometheus Approach)
  9. Push Model (Datadog / InfluxDB Approach)
  10. Decision
  11. 5. System Components
  12. Metrics Agent
  13. Ingestion Service (Metrics Aggregator Gateway)
  14. Kafka (Transmission Pipeline)
  15. Time-Series Database
  16. Query Service
  17. Alerting Service
  18. Visualization Service
  19. 6. Time-Series Database: Storage Internals
  20. Why Not PostgreSQL or MongoDB
  21. Columnar Storage
  22. TSM Files (Time-Structured Merge Tree)
  23. Series Key and Inverted Index
  24. High Cardinality Problem
  25. Storage Tiers
  26. 7. Downsampling
  27. Background Downsampling Worker
  28. Deletion of Raw Data
  29. 8. Alerting Service
  30. Alert Rule Configuration
  31. Alert Evaluation Pipeline
  32. Alert State Machine
  33. Alert Grouping
  34. Alert Inhibition
  35. Alert Routing
  36. Monitoring the Monitor
  37. 9. Query Service
  38. Query Language
  39. Tier Selection
  40. Query Caching
  41. 10. Full Data Flow Summary
  42. 11. Resilience and Fault Tolerance
  43. 12. Technology Choices Summary

1. Requirements

Functional Requirements

  • Metrics collection from diverse sources (OS, database, endpoints, containers, custom application metrics)
  • Metrics transmission from collection agents to the storage layer
  • Time-series storage optimized for high write throughput and time-range queries
  • Query engine for aggregations, rollups, and tag-based filtering
  • Alerting with threshold-based and anomaly-based rules
  • Visualization via dashboards and graphs
  • Data retention with automatic downsampling of older data

Out of Scope

  • Log management (ELK stack territory)
  • Distributed tracing (Jaeger, Zipkin)
  • Incident management and on-call scheduling (PagerDuty)
  • Application Performance Monitoring deep profiling

Non-Functional Requirements

  • High write throughput, 1 million data points per second sustained
  • Low query latency, dashboard queries under 1 second
  • High availability, monitoring systems must never go down (you cannot monitor your monitoring system going down)
  • Horizontal scalability across all layers
  • Data durability, no metric data loss even during broker or storage failures
  • Configurable retention policies per metric

2. What Makes Time-Series Data Different

Every previous system in this series stored business entities: users, posts, orders, files. A metrics system stores a fundamentally different data model.

A single metric looks like this:

text
cpu.usage{host=web-server-01, region=us-east}  72.3  timestamp=1728123456
cpu.usage{host=web-server-01, region=us-east}  71.8  timestamp=1728123466
cpu.usage{host=web-server-01, region=us-east}  74.1  timestamp=1728123476

Three properties make this fundamentally different from business data and require a specialized database:

Property 1 - Time-range query pattern: Every query is a time-range scan. "Give me CPU usage for web-server-01 between 2pm and 4pm yesterday" scans 720 consecutive data points. Standard databases index by primary key or arbitrary columns. Time-series databases index primarily by timestamp, making time-range scans the fast path rather than an afterthought.

Property 2 - Extreme write throughput: Business data writes are sporadic (a user places an order, a post is created). Metric data writes are continuous and uniform: every server emits every metric every 10 seconds, 24 hours a day, 7 days a week. At 1 million writes per second sustained, standard relational databases collapse. Time-series databases use LSM tree-style storage optimized for sequential append writes.

Property 3 - Tiered retention with downsampling: Business data is kept at full resolution forever. Metric data follows a tiered retention model where old data is automatically downsampled to lower resolution to control storage costs. Raw 10-second data becomes 1-minute averages after 7 days. This is a fundamental property of the workload, not an optional feature.

3. Capacity Estimation

MetricValue
Infrastructure monitored1000 server pools × 100 machines per pool = 100,000 machines
Metrics per machine100 metrics (CPU, memory, disk, network, process metrics)
Total unique metric series100,000 × 100 = 10 million series
Collection frequencyEvery 10 seconds per series
Ingestion rate10M series / 10 seconds = 1 million writes per second
Raw data point size16 bytes (8-byte timestamp + 8-byte float value)
Raw storage for 7 days1M writes/sec × 16 bytes × 604,800 seconds = ~9.6 TB
With replication factor 3~29 TB for raw tier, before compression
1-minute tier (30 days)10M series × 43,200 minutes × 16 bytes = ~7 TB per stored aggregate, before compression
1-hour tier (1 year)10M series × 8,760 hours × 16 bytes = ~1.4 TB per stored aggregate, before compression
With Gorilla compression (~11x)Raw tier shrinks to under 1 TB, and every tier shrinks by a similar factor
Total compressed storage across all tiersA few TB per replica, even keeping several aggregates (min, max, sum, count) per window

The storage footprint is manageable. The design challenge is write throughput at 1 million writes per second sustained, not storage volume.

4. Push vs Pull Model

Before diving into architecture, the fundamental design choice: how do metrics get from agents to the ingestion layer?

Pull Model (Prometheus Approach)

The metrics server scrapes each agent at regular intervals via HTTP:

text
Metrics Server -> GET http://web-server-01:9090/metrics (every 15 seconds)
Metrics Server -> GET http://web-server-02:9090/metrics (every 15 seconds)

Advantages: simple agent implementation (just expose an HTTP endpoint), central control over scrape frequency, easy down detection (scrape failure = agent down).

Disadvantages: scraping 100,000 agents from a central server is a throughput bottleneck (large Prometheus setups work around this by splitting targets across many scraper instances), does not work for short-lived batch jobs that finish before the next scrape, agents behind firewalls cannot be reached from outside.

Push Model (Datadog / InfluxDB Approach)

Agents actively push metrics to the ingestion service:

text
web-server-01 Agent -> POST https://ingest.metrics.com/v1/series (every 10 seconds)
web-server-02 Agent -> POST https://ingest.metrics.com/v1/series (every 10 seconds)

Advantages: works behind firewalls (outbound connections only), works for short-lived batch jobs, scales horizontally by adding ingestion servers, agent controls its own push frequency.

Disadvantages: harder to detect silent agents (silence vs down is ambiguous, the same problem as failure detection in distributed systems), agents must know the ingestion endpoint, risk of overwhelming ingestion during cascading failures.

Decision

For this design, the push model is the better choice. Scraping 100,000 agents from a central Prometheus instance creates a single throughput bottleneck. Push distributes load across the ingestion layer naturally and works across network boundaries and firewalls.

The push model's weak spot, a silent agent, is covered with a heartbeat: every agent also pushes a tiny "I'm alive" metric, and an "absent" alert fires when a host that should be reporting goes quiet for a few intervals.

5. System Components

mermaid
flowchart TD
    A[Metrics Agents on 100K hosts] -->|push every 10 s| I[Ingestion Service]
    I --> K[["Kafka: metrics.raw"]]
    K --> DB[(Time-Series DB)]
    DB --> DS[Downsampling Worker]
    DS --> DB
    DB --> Q[Query Service + Redis cache]
    Q --> V[Visualization / Dashboards]
    Q --> AM[Alert Managers]
    AM --> K2[["Kafka: alerts.fired"]]
    K2 --> N[PagerDuty / Slack / Email]

Metrics Agent

A lightweight process running on every monitored host, container, or service. Collects metrics from the local environment and pushes them to the Ingestion Service every 10 seconds.

Collection sources per agent:

  • OS metrics: CPU usage, memory usage, disk I/O, network throughput, system load
  • Process metrics: per-process CPU, memory, file descriptors
  • Container metrics: Docker and Kubernetes container resource usage
  • Database metrics: query latency, connection pool size, cache hit rate
  • Application metrics: custom metrics emitted by the application via a client library (request count, error rate, queue depth)

The agent is intentionally lightweight. It collects, buffers locally for one collection interval, and pushes. It does not aggregate or transform data.

Ingestion Service (Metrics Aggregator Gateway)

Sits between agents and the Kafka pipeline. Receives pushes from 100,000 agents simultaneously.

Responsibilities:

  • Protocol normalization: agents speak different protocols (StatsD, Prometheus exposition format, OpenTelemetry, custom JSON). The Ingestion Service normalizes everything into a standard internal format.
  • Validation: rejects malformed metrics (missing timestamps, invalid float values, metric names exceeding length limits) before they pollute the time-series DB.
  • Tag enrichment: automatically appends standard tags (environment, cluster, datacenter, region) that individual agents should not need to know about.
  • Pre-aggregation: for metrics where exact per-agent values are not needed, computes aggregates at the edge (average CPU across a pool of 100 identical servers) before forwarding, reducing downstream volume. This is opt-in per metric, since a per-host alert needs the per-host values.
  • Rate limiting: prevents misbehaving or compromised agents from flooding the pipeline (rate limiter).

The Ingestion Service is stateless and horizontally scalable. Load balancers distribute agent pushes across Ingestion Service instances.

Kafka (Transmission Pipeline)

At 1 million writes per second, the time-series DB cannot receive writes directly from 100,000 agents. Kafka absorbs the write burst and decouples ingestion rate from storage write rate.

text
Topic:      metrics.raw
Partitions: 100 (partitioned by metric series key for ordering)
Retention:  24 hours (short, just a buffer)

Kafka provides durability during time-series DB slowdowns or restarts. If the storage layer is briefly unavailable, metrics queue in Kafka and are written once storage recovers. No data loss. (Each agent push carries many data points in one message, so 1 million points per second is far fewer Kafka messages than that.)

Time-Series Database

The core storage layer. Covered in depth in section 6.

Query Service

Handles all read requests: dashboard queries, alert rule evaluations, and API queries. Reads from the time-series DB and automatically selects the appropriate storage tier (raw, 1-minute, or 1-hour) based on the requested time range.

Pre-computes and caches commonly queried aggregations (average CPU per service over last 5 minutes) to reduce time-series DB read load from the Alerting Service polling thousands of rules.

Alerting Service

Evaluates alert rules against metrics data and triggers notifications. Covered in depth in section 8.

Visualization Service

Dashboard and graph rendering. Reads from the Query Service. Supports time-series line graphs, heatmaps, bar charts, and single-stat panels. Think Grafana. Stateless and horizontally scalable.

6. Time-Series Database: Storage Internals

Why Not PostgreSQL or MongoDB

In PostgreSQL, each metric data point is a row. Reading one hour of data for one metric series at 10-second resolution requires scanning 360 rows. With 10 million metric series and billions of total rows, random disk seeks for each row are catastrophically slow. Write performance also collapses at 1 million inserts per second on a B-tree indexed table.

Columnar Storage

Time-series databases store timestamps and values in separate columns rather than as complete rows per data point:

text
Timestamps column: [1728123456, 1728123466, 1728123476, 1728123486...]
Values column:     [72.3, 71.8, 74.1, 73.2...]

This enables two critical compressions.

Delta encoding for timestamps:

Timestamps are monotonically increasing and evenly spaced. Instead of storing the full 8-byte timestamp for each point, only the delta from the previous timestamp is stored:

text
First timestamp: 1728123456 (8 bytes, full value)
Delta 2: 10               (1 byte, interval is constant)
Delta 3: 10               (1 byte)
Delta 4: 10               (1 byte)

That's 8x compression on timestamps for regular collection intervals. Gorilla goes one step further and stores the delta of the delta: when the interval stays exactly 10 seconds, the difference between consecutive deltas is 0, which takes a single bit.

XOR compression for values (Gorilla compression):

Developed by Facebook for their internal metrics system. Consecutive metric values tend to be similar (CPU does not jump from 72% to 99% in 10 seconds). XOR-ing consecutive float values produces a result with many leading (and often trailing) zeros, so only the few meaningful bits in the middle need to be stored:

text
72.3 XOR 71.8 = 00000000 00000001... (mostly zeros)
71.8 XOR 74.1 = 00000000 00000011... (mostly zeros)

Combined with delta-encoded timestamps, Gorilla compression achieves roughly 1.37 bytes per data point compared to 16 bytes raw, around 11x compression. Our 9.6 TB of raw data becomes under 1 TB after compression.

TSM Files (Time-Structured Merge Tree)

InfluxDB uses TSM (Time-Structured Merge Tree) files, similar to LSM trees in Cassandra but optimized for time-series data.

Write path:

  1. WAL (Write-Ahead Log): Every incoming data point is first written sequentially to a WAL on disk. Fast sequential append, same as Kafka's commit log. Provides durability if the process crashes before the in-memory cache is flushed.
  2. In-Memory Cache (Write Buffer): Data points are simultaneously written to an in-memory write buffer. Recent reads are served from here without touching disk.
  3. TSM File Flush: When the in-memory cache reaches a size threshold (default 25 MB) or a time limit, it is flushed to disk as an immutable TSM file. TSM files are sorted by series key and timestamp, with columnar storage and Gorilla compression applied. The corresponding WAL entries are deleted after a successful flush.
  4. Compaction: Background compaction merges multiple smaller TSM files into larger ones, applies more aggressive compression, and drops data that has aged past its retention threshold.

Series Key and Inverted Index

Each unique combination of metric name and tags defines a series:

text
cpu.usage{host=web-server-01, region=us-east}
cpu.usage{host=web-server-02, region=us-east}
memory.used{host=web-server-01, region=us-east}

The time-series DB maintains an inverted index mapping tag values to series keys:

text
host=web-server-01  -> [series_1, series_3, series_7, ...]
region=us-east      -> [series_1, series_2, series_3, ...]
metric=cpu.usage    -> [series_1, series_2, series_4, ...]

A query like "give me CPU usage for all hosts in us-east" intersects the metric=cpu.usage and region=us-east lists to find all matching series IDs instantly, then fetches their TSM data. No full table scan.

High Cardinality Problem

The inverted index reveals the most dangerous pitfall in metrics system design. Using high-cardinality values as tags (user_id, request_id, trace_id) causes the number of unique series to explode:

text
cpu.usage{user_id=user_123}  -> 1 series
cpu.usage{user_id=user_456}  -> 1 series
...
100 million users -> 100 million series

100 million series means 100 million inverted index entries. The index alone consumes hundreds of GB of memory and write performance collapses because every new user_id creates a new series and a new index entry.

Rule: only use low-cardinality values as tags (host, region, environment, service_name, cluster). Never use user IDs, request IDs, session IDs, or any unbounded value as a tag. If you need per-user metrics, aggregate them before storage (total_requests_per_minute per service, not per user).

Storage Tiers

The time-series DB maintains three storage tiers:

TierRetentionResolution
Tier 1 (raw)7 days10 seconds
Tier 2 (1-minute)30 days1 minute
Tier 3 (1-hour)1 year1 hour

The Query Service automatically selects the appropriate tier based on the requested time range. A dashboard showing the last 2 hours gets raw data. A dashboard showing the last 3 months gets 1-hour aggregates.

7. Downsampling

Background Downsampling Worker

A background Downsampling Worker runs on a schedule (typically hourly). It reads raw data from Tier 1, computes aggregations, and writes results to Tier 2. It does not run on-demand at query time.

What Gets Computed:

For each metric series and each 1-minute window:

text
Raw 10-second data for cpu.usage{host=web-01} between 14:00 and 14:01:
[72.3, 71.8, 74.1, 73.2, 71.9, 72.8]  (6 data points)

Downsampled 1-minute aggregate:
{
    timestamp: 14:00:00,
    avg:   72.68,
    min:   71.8,
    max:   74.1,
    sum:   436.1,
    count: 6,
    p95:   74.1
}

Multiple aggregations are stored (avg, min, max, sum, count, p95), not just average. This preserves the ability to answer "what was the peak CPU in the last month?" using the stored max value even after raw data is deleted. If only average were kept, peak and percentile queries on historical data would be impossible.

Two details matter when Tier 3 is built from Tier 2:

  • Averages must be weighted. The average of an hour is not the average of sixty 1-minute averages if some minutes had missing data points. Storing sum and count lets the hourly average be computed exactly as total sum divided by total count. (min and max combine trivially: the min of the mins, the max of the maxes.)
  • Percentiles can't be combined at all. The p95 of an hour cannot be calculated from sixty 1-minute p95 values, however you average them. To keep accurate percentiles at every tier, systems store a small mergeable summary of the distribution instead, a histogram with fixed buckets or a sketch like t-digest or DDSketch, which can be added together and still answer "what's the p95?".

Deletion of Raw Data

After the Downsampling Worker writes 1-minute aggregates for a time window, the raw data for that window ages out. Once it exceeds the 7-day raw retention threshold, the TSM compaction process deletes it during its regular compaction run.

text
Day 0-7:   Raw 10-second data in Tier 1
           Downsampling Worker continuously writes 1-minute aggregates to Tier 2
           and 1-hour aggregates to Tier 3
Day 7+:    Raw data ages past retention, deleted during compaction
Day 8-30:  1-minute aggregates (and 1-hour aggregates) available for this time range
Day 30+:   1-minute data ages out
           Only 1-hour aggregates available

The 1-hour aggregates are computed as soon as each hour is complete, long before the 1-minute data expires, so no tier is ever deleted before the next one has its data.

8. Alerting Service

Alert Rule Configuration

Alert rules are stored in a Config DB (PostgreSQL) and cached in Redis on Alert Manager startup:

text
AlertRules
rule_id               UUID, primary key
name                  VARCHAR
metric_query          TEXT (PromQL-style query expression)
condition             ENUM('greater_than', 'less_than', 'equals')
threshold             FLOAT
for_duration          INTEGER (seconds, threshold must be breached continuously)
severity              ENUM('info', 'warning', 'critical')
notification_channels JSONB (slack, pagerduty, email)
group_by              JSONB (tags to group related alerts)
created_at            TIMESTAMP

Alert Evaluation Pipeline

Step 1 - Rule Sharding: 10,000 alert rules are sharded across multiple Alert Manager instances by rule ID. Each instance owns a subset and evaluates them independently. No single instance evaluates all rules.

Step 2 - Pre-computed Rollups: Instead of the Alert Manager querying raw time-series data for every rule every 60 seconds, the Query Service maintains pre-aggregated views of commonly alerted metrics (average CPU per host over last 5 minutes, p99 latency per endpoint over last 1 minute). Alert rules query these pre-computed aggregations, reducing time-series DB read load dramatically.

Step 3 - Threshold Evaluation: Alert Manager evaluates each rule against the pre-computed metric value. If the threshold is breached, it checks the alert state from Redis.

Alert State Machine

The alert KV store (Redis) tracks state per rule per affected entity:

text
Key: alert:{rule_id}:{entity_tags_hash}
Value: {
    state: ENUM('inactive', 'pending', 'firing', 'resolved'),
    breached_at: timestamp,
    fired_at: timestamp,
    last_notified_at: timestamp
}

State transitions:

mermaid
stateDiagram-v2
    [*] --> inactive
    inactive --> pending: breached
    pending --> inactive: recovers early
    pending --> firing: breached for for_duration
    firing --> resolved: back below threshold
    resolved --> inactive: after grace period

The for_duration concept prevents flapping alerts where a metric briefly spikes above threshold for one data point and immediately triggers a notification. The alert only fires after continuous breach for the configured duration. A rule that recovers while still pending goes quietly back to inactive.

Once in firing state, no additional notifications are sent until resolved. last_notified_at enforces re-notification policies (re-notify if still firing after 4 hours).

Step 4 - Kafka Fanout: When an alert transitions to firing, the Alert Manager publishes an alert event to Kafka:

text
Topic:      alerts.fired
Partitions: keyed by the alert's group key (alertname + region + service)

Notification consumers subscribe to this topic and handle delivery to specific channels (PagerDuty, Slack, email, webhook). Keying by the group key (rather than, say, by severity) puts all the alerts that will be grouped together on the same partition, so the same consumer sees all of them and can group them.

Alert Grouping

During a major outage, 500 servers might all breach CPU threshold simultaneously, generating 500 individual notifications. This is useless noise.

The Alert Manager groups related alerts by shared tags before notifying:

text
Group by:        [alertname, region, service]
group_wait:      30s  (wait 30 seconds before sending first notification)
group_interval:  5m   (wait 5 minutes before sending group updates)
repeat_interval: 4h   (re-notify if still firing after 4 hours)

group_wait introduces a deliberate 30-second delay. During an outage, alerts cascade within seconds. The 30-second window collects all related alerts before sending one grouped notification:

text
Before grouping: 500 separate alerts
After grouping:  1 alert "cpu.high on payments service in us-east, 500 instances affected"

Alert Inhibition

Inhibition suppresses lower-level alerts when a higher-level alert is already firing:

text
Rule: if datacenter.down fires for region=us-east
      suppress all alerts where region=us-east

If the entire us-east datacenter is down, individual host-unreachable alerts for all 500 servers in that region are suppressed. Engineers focus on the root cause (datacenter down) rather than 500 symptom alerts.

Alert Routing

Different alerts route to different notification channels based on severity and owning team:

ConditionDestination
severity=critical AND service=paymentsPagerDuty (immediate page)
severity=warning AND service=paymentsSlack #payments-alerts
severity=infoEmail digest (daily summary)

Kafka consumers implement this routing. Each consumer handles a specific notification channel and only processes alerts matching its routing rules. (The Notification System design covers this kind of multi-channel delivery.)

Monitoring the Monitor

The requirements point out that you can't monitor your monitoring system going down, at least not with itself. If the whole system dies, it can't send the alert saying so. The standard answer is a dead man's switch: an always-firing "watchdog" alert that the system sends every minute to an external service hosted somewhere else entirely. That service does the opposite of normal alerting: it pages the on-call engineer when the watchdog alert stops arriving.

9. Query Service

Query Language

The Query Service exposes a PromQL-style query language supporting:

text
# Average CPU across all hosts in us-east over last 5 minutes
avg(cpu.usage{region="us-east"}[5m])

# 99th percentile request latency per service
p99(request.latency{environment="production"}[1m]) by (service)

# Rate of HTTP 5xx errors per endpoint
rate(http.errors{status_code=~"5.."}[5m]) by (endpoint)

(Real PromQL computes latency percentiles from histogram buckets with histogram_quantile, for the same reason as above: percentiles can only be combined across hosts or time through a histogram, never by averaging percentiles.)

Tier Selection

The Query Service automatically selects the storage tier based on the requested time range:

Time rangeTier
Last 6 hoursTier 1 (raw 10-second data)
Last 2 weeksTier 2 (1-minute aggregates)
Last 6 monthsTier 3 (1-hour aggregates)

For queries spanning tier boundaries (last 10 days includes both Tier 1 and Tier 2 data), the Query Service stitches results from both tiers transparently.

Query Caching

Frequently executed queries (dashboard panels that refresh every 30 seconds, alert rule pre-computations) are cached in Redis:

text
Key:   query:{query_hash}:{time_bucket}
Value: query result
TTL:   60 seconds for recent data, longer for historical data

time_bucket rounds the current time to the nearest minute so queries from different dashboard viewers within the same minute hit the same cache entry.

10. Full Data Flow Summary

Metrics Collection Flow:

text
Agent on host -> collects OS, process, container, app metrics every 10 seconds
Agent -> POST https://ingest.metrics.com/v1/series
Ingestion Service:
    -> Protocol normalization (StatsD, OpenTelemetry, custom JSON -> internal format)
    -> Validation (reject malformed metrics and cardinality-violating tags)
    -> Tag enrichment (append region, environment, cluster)
    -> Rate limiting per agent
    -> Publish to Kafka (metrics.raw topic, partitioned by series key)

Storage Write Flow:

text
Kafka consumer -> Time-Series DB writer
    -> Write to WAL (durability)
    -> Write to in-memory cache (fast reads)
    -> When cache full: flush to TSM file (columnar, Gorilla compressed), delete flushed WAL entries
    -> Background compaction: merge small TSM files, drop expired data

Downsampling Flow:

text
Downsampling Worker runs hourly:
    -> Read raw Tier 1 data for elapsed windows
    -> Compute avg, min, max, sum, count, p95 (or a histogram) per 1-minute window per series
    -> Write aggregates to Tier 2
    -> Compute 1-hour aggregates from completed hours, write to Tier 3
    -> Raw data deleted during TSM compaction after 7-day retention

Dashboard Query Flow:

text
User opens dashboard -> Visualization Service
    -> Query Service: parse PromQL expression
    -> Select storage tier based on time range
    -> Redis cache check (query_hash + time_bucket)
    -> Cache hit: return cached result
    -> Cache miss: query Time-Series DB (inverted index -> series IDs -> TSM fetch)
    -> Apply aggregations and mathematical functions
    -> Cache result in Redis (TTL 60 seconds)
    -> Return to Visualization Service for rendering

Alert Evaluation Flow:

text
Alert Manager instances (sharded by rule_id) evaluate rules every 60 seconds:
    -> Fetch pre-computed metric rollup from Query Service (Redis-cached)
    -> Compare against threshold
    -> Check alert state from Redis (inactive/pending/firing/resolved)
    -> If threshold breached:
        -> State: inactive -> pending
        -> If pending for > for_duration: pending -> firing
        -> Publish alert.fired to Kafka
    -> If threshold no longer breached:
        -> State: firing -> resolved
        -> Publish alert.resolved to Kafka

Alert Notification Flow:

text
Kafka alert consumers:
    -> Group related alerts (group_wait 30 seconds)
    -> Apply inhibition rules (suppress lower-level alerts)
    -> Route by severity and service:
        critical -> PagerDuty consumer -> PagerDuty API
        warning  -> Slack consumer    -> Slack webhook
        info     -> Email consumer    -> SES
    -> Update last_notified_at in Redis
    -> Re-notify if still firing after repeat_interval (4 hours)

11. Resilience and Fault Tolerance

  • Kafka as write buffer means the time-series DB can be restarted, upgraded, or temporarily unavailable without losing metric data. Kafka retains 24 hours of metrics. Storage writers catch up from their Kafka offset on recovery.
  • WAL durability in the time-series DB means a crash of the storage process before a flush does not lose data. The WAL is replayed on restart before accepting new writes.
  • TSM immutability means completed TSM files are never modified. Compaction creates new files and atomically replaces old ones. A compaction crash leaves old files intact, no corruption.
  • Alert state in Redis with PostgreSQL source of truth means alert states survive Redis restarts. On Redis cold start, Alert Manager reloads rule configurations from PostgreSQL and rebuilds state by querying recent metric data. Brief state loss during a Redis restart can cause a duplicate notification, or delay a firing alert by up to one for_duration while its pending timer restarts, but it never silently drops an alert.
  • Alert Manager sharding means a single Alert Manager instance failure affects only its subset of rules. Other instances continue evaluating their rules unaffected.
  • Ingestion Service statelessness means any Ingestion Service instance can handle any agent push. Load balancer health checks remove failed instances within seconds. Agents retry failed pushes with exponential backoff and jitter, buffering locally in the meantime.
  • Multi-tier downsampling means historical data is never lost due to raw retention expiry. Downsampling runs continuously and well ahead of raw retention windows. By the time raw data is deleted, its aggregates are already safely written to the next tier.
  • High cardinality enforcement at the Ingestion Service (rejecting metrics with cardinality-violating tags) protects the time-series DB from the most common failure mode that takes down production metrics systems.
  • Dead man's switch to an external service means even a total failure of the monitoring system itself still pages someone.

12. Technology Choices Summary

ComponentTechnologyWhy
Metrics AgentTelegraf / Datadog Agent / Prometheus Node ExporterLightweight host process, push-based, 10-second collection interval
Ingestion ServiceCustom stateless Go or Java serviceProtocol normalization, validation, tag enrichment, rate limiting
Transmission PipelineApache Kafka1M writes/second buffer, 24-hour retention, partitioned by series key
Time-Series DBInfluxDB / Prometheus / VictoriaMetricsTSM columnar storage, Gorilla compression, inverted series index, tiered retention
WALLocal disk append-only log per Time-Series DB instanceDurability for in-memory writes not yet flushed to TSM files
Downsampling WorkerBackground scheduled jobHourly runs, produces Tier 2 and Tier 3 aggregates from Tier 1 raw data
Query ServiceCustom query engine with PromQL supportTier-aware query routing, Redis result caching, pre-computed rollups for alerting
Query CacheRedis with 60-second TTLDashboard query deduplication, alert rule pre-computation cache
Alert Config DBPostgreSQLSource of truth for alert rules, notification channels, routing configuration
Alert State StoreRedisIn-memory alert state machine per rule per entity, fast state transitions
Alert ManagerSharded horizontally by rule_idThreshold evaluation, grouping, inhibition, Kafka fanout
Notification ConsumersPer-channel Kafka consumersPagerDuty, Slack, email, webhook, each independently scalable
Visualization ServiceGrafana-style stateless dashboard rendererReads from Query Service, stateless, horizontally scalable
Meta-monitoringWatchdog alert to an external dead man's switchPages when the monitoring system itself goes silent