Anyone else realize saying yes to customers can slowly change the product?

When I started I wanted the product to stay focused. Every new request felt like something I could probably add without causing much trouble. Then a few small requests turned into a longer roadmap more settings and a product that started trying to do too many things. The hard part is that some of those requests were coming from paying users, so saying no didn’t always feel easy. How do you decide when customer feedback should shape the product and when you need to protect the original idea?