A Beginner’s Guide to Monitoring and Observability

# The practical difference Monitoring tells you the things you predicted would break are broken. Observability is the capability you need for failures you did not predict. Monitoring is cheap and fast for repeated incidents that match your expectations. Observability lets you explore new, surprising failures without changing code first.

# The three signals and when to use them Metrics, logs, and traces each answer different operational questions. Reaching for the wrong one wastes time and money.

• Metrics: Ask "Is something wrong, and how much?" Metrics are numeric, rolled up over intervals, and cheap. Use them to detect changes and drive alerts. They are not suitable for investigating a single request.

• Logs: Ask "What happened in this one case?" Logs are detailed and can contain arbitrary context. They are the right choice when you need per-event detail, but storing and searching logs at scale is expensive.

• Traces: Ask "Where did the time go across services?" Traces follow a request across services and measure latency at each hop. Tracing is usually sampled because full sampling is costly, but it is the only practical way to pinpoint latency sources in multi-service systems.

Working pattern: metrics spot the change, traces locate where time was spent, logs explain what occurred.

# Practical alerting: symptoms, not causes

# Cardinality and the mistake that breaks time-series stores Each distinct label value in metrics creates another stored series. Adding per-request detail as metric labels or exploding label sets with many unique values is the cardinality trap that can overwhelm monitoring systems. Keep metric labels low-cardinality and use logs or traces for high-cardinality, per-request context.

# Instrumentation priorities Start with the four golden signals and basic metrics: request rate, latency, error rate, and saturation. Those signals work for most services and give a practical starting point. Add traces to follow requests across services and logs for root-cause details when needed. If you find you need to deploy a new metric to investigate every unfamiliar incident, your system lacks observability.

# Common real-world failure pattern

# Watchouts and cost trade-offs Do not make logs do metric jobs or metrics do trace jobs. Logging every request to compute latency is expensive compared to using histograms. Alerting on log patterns instead of a numeric metric is slower and costlier. Balance sampling, retention, and the appropriate signal for each operational need.

# One-sentence plan for a team Collect low-cardinality metrics for symptoms, add distributed tracing to locate latency across services, retain structured logs for per-request detail, and design alerts around user-visible impact so responders can act quickly.