How do you make observability a first-class design constraint, not an afterthought?

Category: System Design

Answer

Treat every service as black-box until proven observable: structured logs, request-scoped metrics, traces. ADR every join point. Run a “chaos quarter” each Q where someone pulls the plug and the team measures the diagnosis time. Make the MTTR visible in dashboards, not buried in Slack threads.

Concrete examples from the fca project context

Example 1

Service template emits logs, metrics, traces out of the box.

Example 2

ADR review: every new cross-service call MUST link a metric (Counter) and a trace parent.

Example 3

Chaos drill: kill a dependency; time how long until the team sees the alert + identifies the cause.

Example 4

MTTR is a board-level metric for the team.

Data flow / flow chart

design -> template emits log/metric/trace
         -> chaos drill -> MTTR measured -> iterate

Takeaway

If you can’t see it, you can’t fix it. Bake the sensors into the template, not the maintenance doc.