Johnny Ishak

The hardest bugs to spot in LLM-based apps are the ones where the output looks totally convincing at first glance. I had a bug where the model filled in missing fields ran a search and verified the result except it verified the result against its own generated text rather than the original input.

All my basic tests passed because the output data looked super clean and accurate on paper. It made me realize how dangerous it is to let LLMs modify source data instead of just using them for quick search suggestions.

What kind of checks or boundaries do you write in your code to catch these silent logic bugs?

Visual bugs or broken buttons are easy to spot right away, but silent failures are terrifying because everything looks completely normal on the dashboard.

Our scariest silent bug happened when a background worker quietly stopped processing webhook retries due to a silent timeout exception leaving hundreds of user orders stuck in limbo for days without any logs throwing an alert. We ended up setting up an external health ping just to make sure those queues are actually moving every hour. What's your go to way for catching background failures before users notice?

Spent a lot of time analyzing competitors before launch, but once building started, all those notes became outdated pretty quick.

Tracking everything daily creates useless noise, but waiting months might mean missing major pricing or feature updates. Not sure what's actually worth keeping an eye on long term.

How often do you look at competing products, and what metrics or changes do you totally ignore?