VAmoS Bench
Leaderboard
| Voice agent | Task completion | Connect | Latency | Turns | Barge-in/call | Cost | STT | LLM | TTS | | -------------------- | --------------- | ------- | ------- | ----- | ------------- | -------- | ------------------------- | --------------- | ----------------- | | pipecat | 71.0% | 100.0% | 1.95s | 9 | 0.17 | \~$0.045 | Deepgram nova-3-general | gpt-4.1-mini | ElevenLabs | | livekit | 70.3% | 100.0% | 2.37s | 7 | 0.07 | \~$0.048 | Deepgram nova-3-general | gpt-4.1-mini | ElevenLabs | | vapi | 69.3% | 97.7% | 2.25s | 13 | 0.54 | \~$0.135 | Deepgram nova-3 | gpt-4.1-mini | ElevenLabs | | elevenlabs | 67.3% | 99.7% | 1.19s | 7 | 0.27 | \~$0.114 | Scribe | gpt-4.1-mini | eleven\_flash\_v2 | | cartesia | 67.0% | 100.0% | 2.23s | 7 | 0.02 | \~$0.146 | Ink | gpt-4.1-mini | Sonic | | openai realtime | 67.0% | 100.0% | 1.53s | 7 | 0.15 | \~$0.074 | gpt-realtime-2 | | | | gemini 2.5 native | 64.0% | 99.3% | 15.95s | 7 | 0.01 | \~$0.023 | Gemini 2.5 Native Audio | | | | gemini 3.1 live | 62.3% | 99.7% | 1.38s | 7 | 0.02 | \~$0.016 | Gemini 3.1 Flash Live | | | | retell | 61.3% | 100.0% | 5.76s | 9 | 1.26 | \~$0.208 | Vendor ASR | gpt-4.1-mini | ElevenLabs | | openai realtime mini | 51.0% | 96.3% | 1.79s | 9 | 0.56 | \~$0.032 | gpt-realtime-2.1-mini | | | | nemotron\* | 43.0% | 89.0% | 2.12s | 7 | 0.10 | n/a | Nemotron | Nemotron 3 Nano | Magpie |
Cost vs. completed tasks
45% 55% 65% 75% $0.00 $0.05 $0.10 $0.15 $0.20 Estimated cost per call (USD) · Y: task completion (% of calls) pipecat livekit vapi elevenlabs cartesia openai realtime gemini 2.5 native gemini 3.1 live retell openai realtime mini
Submit an implementation
This is a living benchmark. Every row is one implementation of Riley, the card-ops agent. Build your own on any framework, platform, or model, including ones already on the board. We’ll run it through the same 100 scenarios and publish the results. Feedback on the methodology is just as welcome.