Cekura
Automated QA for Voice AI and Chat AI agents
Start new thread

Cekura Bench - Speech-to-speech model benchmarks on live phone calls

by
•
Cekura Bench publishes voice AI benchmarks you can verify. Our new speech-to-speech benchmark tests 9 realtime voice models, including GPT Realtime 2.1, Gemini Live, Grok and Phonic, as complete phone agents on live calls: 82 scenarios, three runs each. Models are ranked on reliability, data accuracy, stalled calls, response time and cost, and every call transcript is public. Cekura Bench also covers voice agent benchmarks and STT benchmarks, with TTS benchmarks coming soon.

Add a comment

Replies

Best
Maker
📌
Hi everyone, I am Sidhant from Cekura. Today we are launching our speech-to-speech benchmark. We ran 9 realtime voice models as the full agent inside the same open-source Pipecat phone agent, with the same prompt, tools and phone line. Each model handled 82 scenarios across a clinic booking flow and a Medicare intake, three times each. A call passes only if the agent reaches the right outcome and saves every field correctly. Results: Most reliable: GPT Realtime 2.1, 79.3% of scenarios passed on all three runs Fastest: Phonic v1, 1.60s median response Lowest cost: GPT Realtime 2.1 Mini, $0.020 per minute A cascade baseline (Flux, GPT-4.1, ElevenLabs Flash) scored 82.9%, above every realtime model Every call links to its transcript, tool calls and scores, and the agent and test cases are on GitHub. We would value your feedback, especially on which models we should add next.
Maker

Glad to see this live, super useful..

Maker

Most comprehensive benchmarks in voice and it's not even close. 👏

Every speech-to-speech model sounds great in a demo, so we put them all through the same real phone calls with real tool use and scored what actually happened. Excited to see how the rankings shift as new models drop 🚀

Maker

Really interesting results 👀 The cascade baseline beating all the realtime models was definitely not what I expected.

Excited to see what the community thinks - and what models we should put through this next.

Maker

Exciting to actually quantify the explosive growth of voice AI 🚀

Finally, a serious benchmark for speech-to-speech models. No cherry-picked demos. Same agent, same prompt, same tools, and real end-to-end scenarios.

Excited to see what the community does with it. 🚀

Love it! Congrats on the launch team

Publishing the call transcripts alongside the scores is the part I like here. Being able to look at what actually happened on a failed call makes the ranking much easier to judge. Congrats to the team!

Choosing a speech-to-speech model today can be difficult, with new models launching constantly and limited ways to compare them fairly. These results provides transparent, verifiable results to help teams evaluate real-world performance and choose with confidence. 🎯

12
Next