How do you clean up webhook data before it causes problems with your downstream CRM integrations?
Month a partner platform changed their API payload structure without telling anyone turning a regular user ID field into a nested object. That small change broke three days of data across our CRM and email marketing systems.
Of going through thousands of bad records in the database one by one we created a simple validation middleware between our webhook receiver and our internal processing queue. Every incoming payload now goes through a schema check before it reaches any business logic.
If a field does not match the schema we expect the middleware moves the payload to an error log table. Sends a message to our developer channel on Slack right away. This small change has saved us a lot of time that we would have spent fixing data
What error handling strategies do you use to keep your automated workflows safe, from API changes?
Replies
I'd keep rejected events in a replay queue with their event IDs. After fixing the parser, replay through the normal path with duplicate protection so recovery doesn't send the same CRM update or email twice.
A schema check plus a quarantine table is the right base. One layer I added on top, with a warning attached. My backend talks to a music distributor every hour, and a watcher turns the whole feature off by itself when the partner answers in a way it should not: an auth refusal, a paused account, a low balance. It then pings me, and work already sent keeps syncing. The warning: it fired on a perfectly healthy partner, twice. The cause was mine, test fixtures with a fake id made the hourly sync call the partner with ids it never issued, a burst of refusals followed, and the partner briefly refused even the valid token. So the rule became: never call a partner with an id it did not give you, and retry an auth error before believing it.