Why JSON logs beat plain logs at scale?

Category: Observability & Tracing

Answer

Plain logs need regex parsing per line. JSON logs are field-typed: every line has request_id, user_id, agent_step, level, message. A grep for request_id=ABC gives you every log of that one request across all components. Loki / Datadog / CloudWatch all ingest JSON natively.

Concrete examples from the fca project context

Example 1

JSONFormatter: every record is {"ts", "level", "request_id", "user_id", "agent_step", "msg"}.

Example 2

Log handler rotates the file (RotatingFileHandler) for any long-running process.

Example 3

Never log secrets: regex-strip Authorization headers before serialization.

Data flow / flow chart

fastapi middleware -> JSON log {request_id, level, msg, ...}
  Loki / Datadog ingest -> searchable by any field

Takeaway

JSON logs are searchable by any field. Plain logs are searchable by accident.