Building an AI application is only one part of the job. Building an AI application that is scalable, fault-tolerant, observable, and production-ready is what differentiates an AI engineer.
What? FastAPI’s lifespan async-context-manager runs once at startup, yields control to the app, and runs shutdown code on app teardown. It’s where you initialise expensive resources (LLM clients, DB connection pools) and tear them down cleanly.
Why? Without it, every request could leak connections, file descriptors, or LLM clients. With it, your app boots deterministically and shuts down gracefully.
How? Pattern:
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup: build resources ONCE
db_pool = await create_db_pool()
llm_client = build_groq_client()
yield {"db_pool": db_pool, "llm_client": llm_client}
# Shutdown: release things cleanly
await db_pool.close()
await llm_client.close()
app = FastAPI(lifespan=lifespan)Routes access the resources via request.app.state.* or Depends(...).
Scenario 1 — Resource leak without lifespan
A httpx.AsyncClient is constructed inside every route handler. The connection pool leaks 1 socket per request; OOM 4 hours into the day. Fix: instantiate the client in lifespan, attach to app.state, share across requests.
Scenario 2 — Cold-start latency
The first user request takes 8 seconds because pgvector is initialising. Fix: during lifespan startup, send a no-op query to warm the LLM client and the DB pool.
Scenario 3 — Graceful shutdown missing
A rolling deploy kills the pod while long requests are still in flight; clients see TCP RST. Fix: in lifespan exit, await in-flight tasks up to 30s, then pool.close().
What? A multi-stage build that discards build tools from the running layer, keeps only what’s needed at runtime.
Why? Because 2 GB images take 4x longer to start in cluster-autoscale scenarios; egress costs balloon on every pull.
How? Four levers:
slim base (python:3.12-slim): ~150 MB savings vs full python..so and .dist-info to stage 2.--index-url https://download.pytorch.org/whl/cpu saves ~1.5 GB.--no-cache-dir: avoid keeping wheel caches.Scenario 1 — Image is 2.3 GB Diagnosis: CUDA torch + Spacy model + dev tools. Fix: CPU-only torch (200 MB), Spacy wheel in a layer, slim base. Result: 1.3 GB.
Scenario 2 — Cold-start 30s on Cloud Run Diagnosis: 2.3 GB image must download before app can run. Fix: target <500 MB; Cloud Run cold-starts drop to 6s.
Scenario 3 — Image pull failures during k8s rollouts Diagnosis: timeouts on next-mile Google mirror. Fix: pre-pull using a DaemonSet; or mirror to private registry.
What? Multi-cluster or multi-region Kubernetes adds significant network egress. Optimise image, artefact, and request traffic.
Why? Because inter-region bandwidth is expensive (Cloudfront, GCP inter-region); a 1 GB model-image pulled into 50 nodes per cluster = $20 per pull.
How? Three layers:
Scenario 1 — Multi-region production Approach: regional image mirrors; DaemonSet sidecar for embedding models. Each pod reads from local volume, no network per-request.
Scenario 2 — Multi-tenant platform Approach: shared model sidecar; pods share via CSI volume. Saves 1.5 GB per pod × 200 pods = 300 GB of pod-local copies.
Scenario 3 — Edge deployment (Cloudflare Workers or Fastly Compute) Approach: ultra-small model (1B params) + remote LLMs for reasoning. Edge-co-local inference; no inter-region traffic.
What? Operating the same workload across multiple cloud regions introduces latency, data-residency, and consistency challenges.
Why? Because regulatory compliance (FCA, GDPR) requires data to stay in jurisdiction; latency is unfixable across regions; consistency requires design.
How? Three patterns:
Scenario 1 — FCA data residency Approach: per-region Postgres + per-region Redis. UK users only hit UK region. Other regions are dev/staging.
Scenario 2 — Latency-sensitive (real-time agent) Approach: regional user routing based on geo-DNS. Cross-region fallback only on outage.
Scenario 3 — Synchronous global writes Approach: eventual consistency. Writes are async-replicated. Reads can return stale by ~100 ms.
What? Designing for resilience to single-zone outages within a region. Critical because autoscaler responses to a zone failure can deplete a region.
Why? Because zonal failures happen: AWS, GCP, Azure have zone-down incidents every quarter.
How? Three pillars:
minReplicas: don’t autoscale to 0 under load. Triggers cluster thrash.Scenario 1 — Zonal outage during load spike
Effect: one zone’s pods die; HPA spins up 2 zones worth of new pods; cluster-wide outage. Fix: minReplicas: 6 spread across 3 zones; PDB minAvailable: 4.
Scenario 2 — Rolling deploy during high traffic
Effect: 2/3 pods die during deploy, capacity drops below request rate. Fix: PDB minAvailable: 2 ensures 2 pods always run during the rollout.
Scenario 3 — Cascading pod evictions from node pressure Effect: nodes are tight on memory, kubelet evicts low-priority pods. Fix: priorityClass on critical pods; PDB enforces minimum.
What? Async I/O in Python uses cooperative multitasking — explicit await boundaries swap control without OS thread switching. Threading uses OS threads, one per concurrent unit.
Why? Because in production I/O-bound code (LLM calls, DB queries, HTTP), async outperforms threads on cost (one network event loop vs N OS threads) and latency (no thread-pool contention).
How? Rule of thumb:
Scenario 1 — 500 concurrent LLM calls Pick: async. Each LLM call is 1-3s I/O; threads would saturate the GIL and the GIL switch cost is high.
Scenario 2 — Parallel CPU-bound math (matrix multiplication) Pick: processes. Threads would serialise behind the GIL; processes spawn real parallelism.
Scenario 3 — Mixed: 50 I/O calls then a CPU computation Pick: asyncio + asyncio.to_thread for the CPU step. The I/O is concurrent; CPU is offloaded.