LLM Ops Toolkit started as an internal dashboard our AI team used to monitor API uptime and cost drift across models. We open‑sourced it because every AI engineer ends up building something like this eventually.
It combines monitoring, cost analysis, routing simulation, and a maturity audit - all running locally in a static page.
Would love your thoughts on what metrics or integrations you’d want next!
🟢 Provider Status Monitor — Aggregate uptime across 18+ AI API providers with 90-day history and live response-time sparklines 💰 Cost Calculator — True cost analysis including hidden engineering overhead and routing savings 🔀 Routing Simulator — Visualise intelligent request distribution across models with cost/quality trade-offs 📊 Diversity Audit — 10-question maturity assessment with radar chart and personalised recommendations 🌙 Dark/Light mode, auto-refresh, browser notifications for outages, and a dynamic favicon that mirrors overall provider health
Report
LLMs don't 404 - they drift. are you tracking output quality degradation, or just API availability?
@mykola_kondratiuk great thought, would put that on roadmap. For now only latency and uptime.
Report
latency + uptime is an honest start. drift detection gets hard fast - the agent has to expose more of its own state before you can even define what 'drift' means.
Replies
Lamatic.ai
Hello Producthunt Community,
Thanks for checking out our launch!
LLM Ops Toolkit started as an internal dashboard our AI team used to monitor API uptime and cost drift across models. We open‑sourced it because every AI engineer ends up building something like this eventually.
It combines monitoring, cost analysis, routing simulation, and a maturity audit - all running locally in a static page.
Would love your thoughts on what metrics or integrations you’d want next!
🟢 Provider Status Monitor — Aggregate uptime across 18+ AI API providers with 90-day history and live response-time sparklines
💰 Cost Calculator — True cost analysis including hidden engineering overhead and routing savings
🔀 Routing Simulator — Visualise intelligent request distribution across models with cost/quality trade-offs
📊 Diversity Audit — 10-question maturity assessment with radar chart and personalised recommendations
🌙 Dark/Light mode, auto-refresh, browser notifications for outages, and a dynamic favicon that mirrors overall provider health
LLMs don't 404 - they drift. are you tracking output quality degradation, or just API availability?
Lamatic.ai
latency + uptime is an honest start. drift detection gets hard fast - the agent has to expose more of its own state before you can even define what 'drift' means.
Lamatic.ai
@mykola_kondratiuk Its opensource, would you like to collaborate and contribute ?
Happy to take a look. Open-source LLM ops is exactly the kind of infrastructure I'm digging into - I'll check the repo.