Skip to Content

Production Deployment (Rate Limits, Secrets, Streaming)

When to use

Take an agent from notebook to production — handle streaming UIs, rate limits, secrets, observability and graceful shutdown.

Analogy

Moving a band from garage practice to a stadium tour. Way more infrastructure: sound checks, security, ticket gates, streaming, refunds, crowd control.

Data-flow diagram

   FastAPI / asyncio
       |
       v
   agent_runner (LangServe / custom)
       |
       v
   LLM provider + tools + memory + tracing + secrets
       |
       v
   rate limit + circuit breaker + retry + backoff

Deep explanation

Production agents need: streaming for low perceived latency, token-aware rate limiting, secrets management (Vault or env), graceful shutdown (finish in-flight runs, no new ones), circuit breakers (fall back to a simpler model), and per-tenant concurrency caps. LangServe deploys chains as REST endpoints; FastAPI + custom code gives you full control. Always treat the provider as fallible.

Examples

Example 1

# LangServe: serve a chain as REST + streaming
from langserve import add_routes
from fastapi import FastAPI
app = FastAPI()
add_routes(app, chain, path='/agent')
# client streams via SSE
import httpx, json
async with httpx.AsyncClient() as c:
    async with c.stream('POST', 'http://localhost:8000/agent/stream',
                        json={'input':{'topic':'cats'}}) as r:
        async for line in r.aiter_lines(): print(line)

LangServe wraps a chain in REST + SSE; clients can stream tokens as they arrive.

Example 2

# token-aware rate limit
from limits import TokenBucketRateLimiter
limiter = TokenBucketRateLimiter(storage)
def rate_limit(user_id, n_tokens):
    return limiter.consume(user_id, n_tokens) if limiter else True

Token-bucket rate limit per user caps spend independent of request count.

Example 3

# circuit breaker: fall back to a smaller model on outage
from langchain_anthropic import ChatAnthropic
robust = primary.with_fallbacks([ChatAnthropic(model='claude-haiku-4-5')])
robust = robust.with_retry(stop_after_attempt=3, wait_exponential_jitter=True)

with_retry + with_fallbacks combine for resilient production LLM calls.

Common mistake

Running agents in a single process without graceful shutdown. SIGTERM kills in-flight tasks; users see partial answers.

Key takeaway

Stream for UX; rate-limit per token; bind secrets from Vault/env; use retries + fallbacks; implement graceful shutdown; cap concurrent runs.

Production Failure Playbook

Failure scenario 1: secrets-in-source

Failure scenario 2: graceful-shutdown-missing