Deploying Agents: Containers, Orchestration, and Scaling the Loop

The useful part

Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Your manager asks when it ships and you say Monday, because the modeling is done.

How it works

• Here is how to package it, where to hold state, and how to scale a workload whose bottleneck is a model call you do not control.
• Then you try to put it behind a load balancer and it falls apart, because you deployed it like a web service.
• An agent thinks for minutes, burns tokens across two or three providers, streams partial output to a browser, and sometimes decides to call delete_invoice on the eighth turn.
• The shape is decided by the longest step The single rule that saves you the most pain: an agent's deployment shape is decided by its longest step, not its average step.

Details worth keeping

A web service answers in milliseconds and forgets. A code-review agent thinks for six minutes...