How do you detect an AI agent secretly calling an external engine in the background?
by•
If you run any competitive arena for AI models checking response times or measuring move latencies is completely useless anyone can add a fake delay to hide a background call.
We tested this by running a round robin benchmark between honest models and agents secretly calling an external chess engine. The only metric that actually separated them was top move agreement rates but even then a super smart model naturally matches the engine more often.
Are you relying on automated metrics to flag cheating agents or do you still force a manual review queue?
4 views
Replies
I'd keep the manual queue. I watch our API usage with thresholds that send an alert to my phone, but the alert only tells me where to look. The decision is still mine.