System design is where AI roles get paid the most. Knowing the model isn’t enough; you need to know how to architect, deploy, monitor, and scale around it.
Real AI engineering is about building systems that are measurable, reliable, grounded, cost-aware, and useful in production. Models are powerful, but the system around the model determines whether the product actually works.
What? A multi-region chatbot that handles 100k conversations/day with sub-2-second latency and <0.5% hallucination rate.
Why? Because this is the canonical “design me a system” prompt for AI roles. The interview passes if your architecture handles scale, cost, and observability simultaneously.
How? Build it from components you already understand:
Scenario 1 — Burst traffic (Black Friday support surge) 10x normal load for 4 hours. Architecture choices: HPA on CPU; pre-warmed worker pool; queue-based decoupled ingest; autoscale Redis + Postgres read-replica. Without these you 500 the world.
Scenario 2 — Latency target < 200ms for first token Architecture: Tier-0 Redis cache for popular questions; streaming via SSE; model warm-pool (pre-loaded weights); parallel routes (the supervisor decides while the LLM is already responding). Reduces perceived latency without lowering answer quality.
Scenario 3 — Multi-tenant (50 fintech clients in one cluster) Architecture: per-tenant vector collections (or per-tenant postgres schemas); strict tenant filter on every query; tenant header propagated from auth-token; per-tenant rate limits.
What? A cost-engineering mindset: token budget, model-tiering, caching, batching, and observability.
Why? Because cloud bills can dwarf salaries in a runaway AI app.
How? Five levers:
Scenario 1 — Grew 10x in 3 months (cost went from $5k to $80k/month) Diagnosis: no caching, GPT-4 for every intent. Fix: add Redis cache for FAQs and common intents; classifier runs on 8B open-source model; GPT-4 only for complex multi-hop reasoning. Cost: $80k → $25k.
Scenario 2 — LLM token usage growing unbounded Diagnosis: conversation history is sent in full. Fix: truncation with token counting; keep last N turns; summarise older turns server-side.
Scenario 3 — Users complain about response depth but cost is rising Diagnosis: high temperature + long context = many tokens used. Fix: shorten context; raise concurrency; use long-context models only when the conversation requires it.
What? A continuous-evaluation pipeline where every conversation is scored for quality, retrieval accuracy, and adherence to rules.
Why? Because “looks good last week” is not “good today.” Models drift, embeddings drift, intent distribution shifts.
How? Three layers:
Scenario 1 — Quality drop after re-training a sub-component Diagnosis: nightly eval catches the regression in 24 hours. Without it, the team learns two weeks later via twitter anger.
Scenario 2 — New product launched, intent distribution shifts Diagnosis: intent classifier is now seeing 30% “pension” intents it never saw before. Confidence drops. Prompt-team adds few-shot examples; nightly eval confirms restored quality.
Scenario 3 — Vendor model deprecation Diagnosis: nightly eval catches the silence from one model. Migration: baseline “old model” answers in your golden set; pin to new model + compare against golden set.
What? Defences against prompt injection, jailbreak, PII leakage, and runaway costs.
Why? Because the model surface is new attack surface; classic web defenses don’t apply.
How? Six layers:
Scenario 1 — Customer types a phishing-style prompt
Pre-LLM: Lakera flags. Pre-LLM filter returns {”is_injection”: true, ”confidence”: 0.95}. Request is logged, customer is asked to rephrase.
Scenario 2 — LLM emits PII its own training-data was scraped from
Post-LLM: Presidio detects john.doe@gmail.com in the response. Substitute with <EMAIL>. Log the swap.
Scenario 3 — Adversarial customer loops an agent for $900 of LLM calls Cost-facing: per-session token tracker. After 100k tokens, abort the loop. Detective layer: loop detector sees same intent for 50 turns, force-escalate to human.
What? When you have no production traffic yet, how do you know if the system is safe?
Why? Because most AI rollouts fail at this exact phase. They optimise the model but skip the rollout muscle.
How? “Dark launch + canary + ramp”:
Scenario 1 — Banking chatbot replacing IVR (“Press 1 for…”) Approach: shadow-run for 2 weeks; canary at 5% to engaged technical customers; observe escalation rate and CSAT. Ramp to 50% if metrics hold.
Scenario 2 — Internal AI summarisation tool Approach: shadow-run; sample outputs from a few hundred documents and grade against human summaries. No production risk because users aren’t affected.
Scenario 3 — Customer-facing search-AI Approach: A/B test with ~5% traffic; monitor satisfaction (thumbs up/down), fallback rate, latency. Kill switch via feature flag.