LangWatch Optimization Studio - Evaluate & optimize your LLM performance with DSPy

Hello Product Hunters!! 🚀 I’m Manouk, co-founder of LangWatch. You may haven't heard from us in a while, but trust me—you’ll love this new ProductHunt launch. We’ve been listening and have solved your biggest pain points in building LLM applications. 🥁 And the best part? We have opened-up the access to all users. We’re thrilled to be live on Product Hunt today with our biggest release yet: LangWatch Optimization Studio! After building our powerful Monitoring & Evaluation platform, there was still a major pain point you shared with us. So, we solved it: This new feature takes LLM development to the next level, solving the most frustrating challenges teams face when optimizing LLM pipelines and ensuring top-tier performance. With the newest Optimization Studio, you can: ⚡ Evaluate & Optimize: Experiment different prompts and LLM-models automatically in seconds (with DSPy) 🧠 Custom Evaluations: Build tailored evaluations and bring them back to your real-time monitoring. 🔄 Seamless Deployment: Confidently push optimized prompts/few-shot examples and pipelines into production. A few highlights of LangWatch: 🌟 Observability: Monitor LLM performance, latency, costs, and quality. 📊 User-Analytics: Deeply understand what docs of your RAG are mostly used and topics discussed. 👩‍💻 Evaluations: Run evaluations to control for hallucinations or other criteria you care about real-time. ✏️ Prompt Management: Version, tweak, and deploy prompts directly from LangWatch. 🔬 Datasets: Build datasets of non-performing traces and start improving. 🐒 API: All analytics and features available per API. 🌍 Open Source: Join our growing community on GitHub to collaborate and contribute. Get started today: ⭐ GitHub: https://github.com/langwatch/lan... 📖 Docs: https://docs.langwatch.ai/ ⏯️ Live Training Today: https://lu.ma/um4owj65 If you're already a fan of DSPy, we'd love to hear about what you've built and explore how we can support you even more! A big thanks to the PH community for all your feedback and support. We’re here all day and can’t wait to hear your thoughts, questions, and feedback! Cheers, Manouk

Have worked with DSPy before, very curious towards how you guys are doing this.. How do I build a custom evaluation?

Really enjoyed working on this project, still a lot to offer. We will keep on adding to it, as the use cases are really infinite!!

If you are an AI engineer in the industry for a while, you know that handling quality is the biggest challenge for developing with LLMs, and we are here to change that, we promise you won't regret trying it out! We've been head on working with AI engineers on the best way to guarantee LLMs behave as you want, that you can measure what quality means, experiment super fast and then use the DSPy optimizers to automatically find that prompt that will maximize the quality. Excited to be live on Product Hunt with the Optimization Studio, we think you are going to love it. Hit me up for any questions on the product!

Congrats on launching LangWatch! Looks like an awesome tool to help AI developers work smarter and faster. Wishing you lots of success!

Very interesting feature release! 🔥

Seems like a great product for lllm eval, keen to give it a try

Working with LLM's daily, I welcome solutions to help with Quality Assurance and Testing. Nice job!

This is exactly what LLM developers need! 🎯 The RAG context tracking feature is especially interesting. How granular can we get with the custom evaluations for production monitoring?

🔥🔥🔥