api: YAML JSON TOON Database
version: 1.0.0
format: yaml
dataset:
  id: 1633
  slug: observability-signals
  title: Observability Signals and Alert Design
  description: Operational signals, useful questions, and failure modes for service monitoring and incident diagnosis.
  summary: Operational signals, useful questions, and failure modes for service monitoring and incident diagnosis. Each row includes a practical application and a failure to check before using it.
  use_when:
    - Add useful diagnostics to a small API.
    - Choose alerts tied to a real user-visible failure.
  caveats:
    - Names and conventions vary by instrumentation stack.
    - Collecting telemetry does not by itself provide an actionable alert.
  related:
    - site-reliability-practices
    - reliable-background-jobs
  reviewed_at: "2026-10-12"
  sources:
    - title: OpenTelemetry signals
      url: "https://opentelemetry.io/docs/concepts/signals/"
    - title: Google SRE monitoring guidance
      url: "https://sre.google/sre-book/monitoring-distributed-systems/"
  reading:
    - logs-metrics-and-traces-for-a-small-api
  provenance: Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
  entry_count: 10
  fields:
    - section: data
      field: signal
      type: string
      example: Metric
    - section: data
      field: question
      type: string
      example: How much or how often?
    - section: data
      field: example
      type: string
      example: Request count per service and status class.
    - section: data
      field: pitfall
      type: string
      example: Unbounded labels create excessive cardinality.
  category: DevOps
  category_slug: devops
  tags: devops,observability signals,reference,engineering
  view_count: 15
  created_at: 1791798640
  updated_at: 1791798640
data:
  - signal: Metric
    question: How much or how often?
    example: Request count per service and status class.
    pitfall: Unbounded labels create excessive cardinality.
  - signal: Log
    question: What happened at this event?
    example: A structured failure record with an operation ID.
    pitfall: Logging credentials or full private payloads.
  - signal: Trace
    question: Where did a request spend time?
    example: Spans covering a request and its database query.
    pitfall: Missing propagation breaks the end-to-end path.
  - signal: Latency histogram
    question: What does the duration distribution look like?
    example: Bucket counts for request duration.
    pitfall: A mean hides slow-tail behavior.
  - signal: Error ratio
    question: What fraction of eligible requests failed?
    example: Failed requests divided by all eligible requests.
    pitfall: A raw error count changes with traffic.
  - signal: Saturation
    question: How close is a resource to its capacity?
    example: Queue depth relative to workers.
    pitfall: A high utilization number alone may not indicate user harm.
  - signal: SLI
    question: What measured behavior represents service quality?
    example: Fraction of requests completed within the latency target.
    pitfall: The denominator must match user-relevant work.
  - signal: SLO
    question: What reliability target do we intend to meet?
    example: A target success fraction over a defined window.
    pitfall: A target without a window is underspecified.
  - signal: Burn rate
    question: How quickly is the error budget being consumed?
    example: Observed error fraction divided by the allowed fraction.
    pitfall: Short windows can be noisy at low volume.
  - signal: Correlation ID
    question: Which records describe one operation?
    example: A stable request or job identifier across signals.
    pitfall: It is not a user identity and should not contain secrets.