[Private] Beyond Benchmarks

A Closed-Door Dinner on Evaluating Models in Production

This is not a public event.

On May 14th, AI Tinkerers and OpenRouter are convening a small group of builders for a private VIP dinner in San Francisco. Attendance is by invitation only.

No applications. No +1s. No observers.

Each guest has been hand-selected for their direct experience shipping and operating AI systems in production.

The Theme: When Benchmarks Stop Mattering

Benchmarks are clean. Production is not.

With the rapid iteration of frontier and open models, leaderboard performance has become increasingly disconnected from real-world reliability. The teams actually deploying these systems are forced to build their own evaluation loops-grounded in latency, cost, user behavior, and failure modes that don’t show up in static tests.

This dinner is a closed discussion among those actively navigating that gap.

We will focus on:

• Live Evaluation Systems: How teams are instrumenting real user traffic to continuously evaluate model performance.
• Routing as a Primitive: Using gateways to dynamically select, fallback, and optimize across models in production.
• Model Drift in Practice: Detecting and responding to silent regressions as providers update weights and infra.
• Cost vs. Reliability Tradeoffs: When cheaper models break-and when expensive ones aren’t worth it.

Format

There are no presentations.

No slides. No intros. No spectators.

Just a single table of builders exchanging what’s actually working-and what isn’t.

• Hand-Selected Room: Every attendee is actively shipping AI systems at a high level.
• Off-the-Record: This is a confidential environment. What’s shared in the room stays in the room.
• Operator-Only Dialogue: We skip strategy and focus entirely on implementation, edge cases, and failure modes.

About the Sponsor

OpenRouter is the largest, most battle-tested AI model exchange: 300+ models from 60+ providers, processing ~30 trillion tokens monthly. Instead of hand-rolling routing logic and juggling provider keys, you route through OpenRouter once and let Auto Router pick the best execution path for every request - with automatic failover when providers degrade.

AI Tinkerers is a curated community of active builders. Events like one this are not open to the public.