Arbiter -- LLM Evaluations with automatic interaction tracking, multiple evaluators, and extensible architecture

I’ll demo Arbiter’s automatic observability for AI agents using two examples that show what makes it different: you get complete visibility into LLM calls and costs with zero instrumentation.

Live demo (5-10 min):
Example 1: Interaction Tracking (interaction_tracking_example.py)

• Run a simple evaluation (semantic similarity check)
• Show the automatic interaction log that appears - every LLM call captured with:
  • Exact prompts and responses
  • Token usage (input/output)
  • Latency per call
  • Model used
  • Timestamp
• The key point: You didn’t instrument anything. This just appears automatically.

Example 2: Cost Tracking (cost_tracking_example.py)

• Same evaluation, now show automatic cost calculation
• Breakdown by evaluator, by model, by token type
• Uses llm-prices dataset (100+ models, kept up-to-date)
• Show how this scales: batch evaluation cost analysis across multiple outputs
• The key point: Know exactly what each evaluation costs before you scale to thousands of agent interactions

Why this matters for agent builders: When you’re running 10K agent interactions daily, you need to know what’s happening and what it costs. Arbiter gives you both automatically. No logging infrastructure to build, no cost tracking to implement - it’s just there.