Shipping is the start of the expensive part. A feature that works on the day it launches and degrades quietly over the following month has cost more than it delivered, and this is a common outcome for AI features in particular because their failures do not raise errors.
What the work involves
Deployment that is boring. Automated, repeatable, and reversible. If releasing is stressful, releases become rare, and rare releases become large, which makes them riskier. The loop reinforces itself in whichever direction it starts.
Observability that answers questions. Not dashboards for their own sake, but the specific signals that let you find a cause quickly: structured logs with enough context to trace a request, metrics on the things that actually degrade, and alerts that fire on user impact rather than on noise.
Output quality monitoring. Specific to systems with a model in them. Error rates stay flat while answers get worse, so sampling real output and scoring it is the only thing that catches quality decay before users report it.
Cost as an operational metric. Tokens per request, calls per interaction, and spend per active user belong on a dashboard next to latency. Cost problems found on an invoice are always found late.
Designing for dependency failure. Every external service is occasionally unavailable, slow, or subtly different than it was last week. Timeouts, budgets, retries with a ceiling, and a defined behaviour for when the dependency is gone are part of the feature, not an afterthought.
How I approach it
Assume it will break and decide in advance what happens when it does. A predictable degraded experience beats an unpredictable full one.
Pin versions, including model versions. A hosted model called by a floating name is a dependency that can change without your involvement.
Watch the tail, not the average. Users experience the slow requests, and an average hides them completely.
Keep a record of why things are the way they are. Most production incidents are extended by somebody not knowing why an odd looking decision was made, and removing it.
Related reading: why AI features fail in production and the real cost of a long context window.