Verify LLM API - Check which model your AI agent is really using
by•
Verify LLM API checks whether the model behind your AI agent or API matches its claim. Tell a Codex agent to run verifyllmapi.com/run, or test an OpenAI- or Anthropic-compatible endpoint. It reuses the Agent’s login in fresh sessions, samples one-token outputs, and matches their distribution with public fingerprints using Jensen–Shannon distance. Credentials and raw samples stay local. The library covers 161 exact model IDs; unknown or weak matches return INCONCLUSIVE.
Replies
The privacy choice of keeping credentials and raw samples local is genuinely thoughtful, especially for a tool that has to handle API keys to do its job.
@duraloguzh72806 Thank you — that boundary was non-negotiable for me. One small clarification: in current-Agent mode the Skill asks the host to run fresh sessions; it never reads login files or tokens. For an explicit API endpoint, the scanner reads the key locally from an environment variable or env file and sends it only to that endpoint. VerifyLLMAPI never receives the key or raw samples.
That constraint adds some friction, but it keeps the trust surface small. If you spot anywhere the docs make this boundary unclear, I’d genuinely like to fix it.
ran it on a couple of internal endpoints and it caught one Claude ID that was secretly routing to a smaller model, which honestly surprised me. The INCONCLUSIVE fallback is smart too.
@songlsankxxlh That’s a useful catch — and also a good case for being careful with the wording. I’d treat the mismatch as a lead to inspect the routing logs, not proof on its own that the provider swapped the model. Wrappers, model updates, and decoding settings can move a fingerprint too.
If you still have the local report, I’d be curious about the claimed ID, the nearest reference, and whether the reported route stayed stable across the full run. Please redact endpoint names and anything sensitive. And yes, INCONCLUSIVE is deliberate: weak evidence should stay weak evidence.
the fingerprinting approach is clever, especially keeping credentials local. one thing i'd love to see is a way to schedule periodic verification on a cron or webhook basis, so i can get alerted if my api provider silently swaps the model under me. that would make this way more useful for production monitoring.
@aslanbaynu67506 Yes — that’s the step that turns this from one-off verification into monitoring. The CLI already supports machine-readable --json, so today it can sit inside your own cron or CI job while the credentials stay local. What’s missing is the product layer: a saved baseline, sensible change thresholds, history, and a webhook only when the evidence is strong enough.
I’m cautious about running the scheduler as a hosted service because that would ask users to trust another system with endpoint credentials. My preferred design is a small local runner that stores the baseline and sends a redacted alert. Your comment is a strong vote to make that a first-class flow. Would cron plus a generic webhook cover it, or would GitHub Actions be useful too?
Model routing is getting increasingly opaque. Curious what was the original pain point that convinced you this needed to exist as a standalone product?
@aryan787544 That was exactly it: the model name in a config or API response is still just a claim. I was testing agent setups where provider aliases and routing layers made it hard to tell what actually answered, and the failure mode was silent — everything still worked.
I wanted one small local check that reused the existing login, kept keys local, and was willing to say INCONCLUSIVE instead of guessing. It became a standalone tool when I saw the same problem across Codex sessions, OpenAI-compatible APIs, and Anthropic-compatible APIs. Coverage is still the hard part, but the need felt broader than one integration.