← Home
TWIML · April 30, 2026 · 54m

How to Engineer AI Inference Systems with Philip Kiely - #766

Philip Kiely
Show Notes

Philip Kiely, head of AI education at Baseten, discusses inference engineering as the stickiest and most critical AI workload. The conversation covers the blend of GPU programming, applied research, and distributed systems that define inference, and how teams can now go from research to production in hours rather than months. Philip explains the key knobs—batching, quantization, speculative decoding, and KV cache reuse—that let teams balance latency, throughput, and cost. He traces the maturity journey from closed APIs to dedicated deployments and in-house platforms, surveys the runtime landscape (vLLM, SGLang, TensorRT LLM), and looks ahead to the need for specialized, workload-specific runtimes as agents and multimodality demand ever more efficient inference.

This summary was generated from show notes and public descriptions, not from a full transcript review. Details may contain inaccuracies.

Highlights

•
Inference as the Stickiest AI Workload
Inference has become the stickiest and most critical workload in AI, blending GPU programming, applied research, and distributed systems.
ClaudeChatGPTGemini
•
Research-to-Production in Hours
With modern inference tools, teams can move from research breakthroughs to production deployment in hours rather than months.
ClaudeChatGPTGemini
•
The Inference Knobs: Batching, Quantization, Speculation, KV Cache Reuse
Key inference optimizations—continuous batching, quantization, speculative decoding, and KV cache reuse—allow teams to tune latency, throughput, and cost.
ClaudeChatGPTGemini
•
Inference Maturity Journey: Closed APIs to In-House Platforms
The typical path for AI inference matures from using closed model APIs, to dedicated managed deployments, and eventually to building in-house serving platforms for performance and control.
ClaudeChatGPTGemini
•
Runtime Landscape: vLLM, SGLang, TensorRT LLM
The inference runtime landscape includes vLLM (focusing on open-source efficiency), SGLang (structured generation), and NVIDIA’s TensorRT LLM (deep optimization for NVIDIA GPUs).
ClaudeChatGPTGemini
•
Specialized Workload-Specific Runtimes for Agents and Multimodality
For demanding emerging workloads like agents and multimodal models, specialized inference runtimes will be necessary to achieve the required performance and efficiency.
ClaudeChatGPTGemini
Was this useful?