I Built a JIT Compiler for AI Agents: How We Turned 30s LLM Chains into 0.1ms Deterministic Python

The useful part

In 2026, AI agents have become the default paradigm for automating complex workflows: DevOps orchestration, customer support, database triage, and e-commerce transactions. Crippling Latency: A standard 4-step tool chain (think -> tool -> observe -> think) easily burns 15 to 45 seconds.

How it works

• Here is the unspoken truth of agent workflows: over 90% of recurring invocations execute the exact same sequence of tool calls, differing only by input parameters (e.g., user_id, order_id, or date).
• Why are we invoking massive 70-billion-parameter neural networks over HTTP dozens of times just to parse an ID and pass it into a database query?
• Running that loop 50,000 times a day costs thousands of dollars every month for redundant reasoning.
• Even with a 98% success rate per step, a 4-step chain has an ~8% failure rate.
• Interpret dynamic code on the first run, profile hot execution paths, and compile them into native machine code.

What to take from it

Trace dynamic tensor operations and fuse them into deterministic CUDA/C++ kernels. Today, I’m open-sourcing AgentJIT â€" a Just-In-Time trajectory compiler for AI agents that trace...

Details worth keeping

What if we did the exact same thing for AI Agent Trajectories?