Does anyone else feel like giving coding agents more context actually makes them lose focus?

I thought feeding the entire repo to the agent would make it smarter. Instead, it starts getting distracted by legacy code and tries to "clean up" stuff completely unrelated to my prompt.

you literally ask for a button color change and it delivers a pull request touching twelve different files.

How do you guys restrict the working directory or keep the agent focused on just one file?