While building WAIL, one problem kept coming up: infrastructure-level failures are easy to recognize when a provider returns a timeout or 5xx. The harder cases are requests that technically succeed.
A request can return 200 OK while first-token latency has increased significantly, throughput has degraded, or the stream has become unstable. From the application's perspective the call succeeded, but operationally the execution may already be unhealthy.
The approach I ended up taking with WAIL was to establish runtime baselines, evaluate degradation while execution is happening, and separate observation from control. Depending on the situation, WAIL can observe, retry, or reroute, while recording signed evidence of the decision and outcome. It runs alongside the provider client rather than requiring a gateway.
I'm curious how other teams running AI systems in production draw this line:
WAIL is an on-prem AI runtime control and governance layer. It detects unhealthy model behavior and can retry or reroute when intervention is justified. Decisions can apply to the next request, while entitled plans can reroute in-flight on TTFT degradation before the first token. WAIL records signed evidence of what happened and why. It wraps your existing provider SDK; it is not a gateway. Prompts, responses, and API keys stay in your environment.
