Deploying AI infra at scale using adaptive load balancing

This session walks through adaptive load balancing for LLM traffic using live code and running services. I’ll open up the Bifrost gateway internals (open source, written in Go) and walk the audience through the request lifecycle: routing, health scoring, concurrency control, quota enforcement, and failover logic.

We’ll look directly at:

• How the router scores providers and regions based on live latency, error rate, and budget usage.
• How per-tenant rate limits and spend controls are enforced without blowing up p95.
• How fallback routing and provider failover are implemented so apps keep responding even when one model endpoint degrades.

I’ll then run a small cluster, generate load, and intentionally degrade one upstream provider. We’ll watch how traffic shifts in real time, inspect traces in Grafana, and read the routing decisions back from telemetry.

Attendees leave with the control loop in Go, the deployment shape (Kubernetes + Postgres + OTEL), and a mental model they can reapply in their own infra.