Observability Guide

Observability Guide

MANDATORY FOR ALL AI AGENTS: Every new use case, adapter, or entry point MUST include appropriate observability instrumentation. Metrics MUST be added via decorator adapters (Chain of Responsibility pattern), NEVER inline in business logic. Tracing is provided automatically via Micrometer-OTEL bridge. Logging is correlated via MDC populated by the tracing bridge.

Project-agnostic: code samples use placeholder package com.example.platform, service name platform-service. Replace with your project’s actual values when applying patterns.

This file is the index. Concrete patterns live in the four themed files below.

Companion guides:

• hexagonal-architecture.md — Chain of Responsibility / decorator pattern.
• security.md §10 — what NEVER to log (PII, secrets).
• adr.md — capture observability backend decisions (Prometheus vs DataDog vs NewRelic).
• arch-doc-spec.md §Observability — NFR + SLO section.

Companion command/agent: /arch-otel → arch-observability-injector (generates metrics decorator wrapping a port).

Themed Files

1. Observability Architecture

The three pillars of observability are implemented as cross-cutting concerns, fully decoupled from business logic via the Chain of Responsibility (Decorator) pattern.

Three Pillars

Core Principle

Observability is a cross-cutting concern and MUST be handled via the decorator pattern (Chain of Responsibility). This is defined in hexagonal-architecture.md under the “Chain of Responsibility Pattern (Decorator Pattern)” section.

Hard rules:

• Business logic (use cases, domain) MUST NOT contain any metrics, tracing, or observability code.
• Metrics adapters live in the metrics-publisher infrastructure module.
• Metrics adapters extend MetricRecorded and implement the corresponding port interface.
• The chain is wired in UseCaseConfig (bootstrap module) via buildChain().
• Tracing is automatic — the Micrometer-OTEL bridge instruments spans without code changes.
• Logs are correlated via MDC fields (trace-id, span-id) populated by the tracing bridge.
• Never log secrets, tokens, PII, or full request bodies. See security.md §10.

Architectural Flow

1
Entry Point (REST / Kafka)
2
    │
3
    ▼
4
[Metrics Adapter]  @Order(10)   → records timer, counter, error rate
5
    │
6
    ▼
7
[Cache Adapter]    @Order(50)   → optional cache lookup
8
    │
9
    ▼
10
[Use Case]         @Order(100)  → pure business logic (no observability code)
11
    │
12
    ▼
13
[Repository Chain]
14
    [Cache]    @Order(50)
15
    [Metrics]  @Order(10)
16
    [JDBC]     @Order(100)

All adapters in the chain are linked via ChainablePort<T>.setDelegate() and sorted by @Order in UseCaseConfig.buildChain().

Reading Order

Discovering observability (first read):

1. This page — architecture + three pillars.
2. obs-metrics.md — how custom metrics integrate via decorator.
3. obs-tracing.md — how tracing flows automatically through.
4. obs-logging.md — how logs correlate with traces.
5. obs-recipes.md — apply to a new feature end-to-end.

Adding observability to a new feature:

1. obs-recipes.md §9 — 5-step recipe.
2. obs-metrics.md §5 — pick metric names + tags.
3. obs-logging.md — structured log fields.
4. obs-recipes.md §10 — verify with Prometheus + Grafana.

Production troubleshooting / on-call:

1. obs-recipes.md §10 — key metrics, alerting rules, queries.
2. obs-tracing.md §NewRelic NRQL — trace queries.

Pre-Merge Checklist (delta)

For PRs that add a new use case, adapter, or entry point:

• Metrics decorator implemented and registered in buildChain() (see obs-recipes.md).
• No MeterRegistry injected into use case or domain.
• Metric names follow conventions (see obs-metrics.md §5).
• Tags use the documented allowlist (no unbounded cardinality — never userId as a tag).
• Structured logs added at boundaries with traceId, userId, tenantId (see obs-logging.md).
• No PII / secrets in logs (see security.md §10).
• Dashboard updated in Grafana (or template generated) when a new SLO is introduced.
• Alerting rule reviewed for new critical metric.

Full checklist in obs-recipes.md.