Guide / AI & Automation
AI Infrastructure Starts With Boring Questions About Cost, Latency and Failure
Latency, cost, rate limits and recovery shape an AI feature long before a model benchmark settles the choice. Design the dependency around the actual user task.
Before choosing an AI stack, I want to know what happens when the model takes too long. Does the user wait? Can the task continue in the background? Is a partial result useful? Does another request repeat an expensive operation, or resume the same one?
These questions sound less exciting than model comparisons, but they describe the infrastructure a feature actually needs. A model can perform well on an evaluation and still be a poor fit for a particular interaction. The surrounding system determines whether its behavior becomes useful, affordable, and recoverable.
Consider a hypothetical tool that reads a support conversation and prepares a draft response. It does not send the response automatically. Even within that modest boundary, the system needs decisions about waiting, context size, customer information, retries, and review.
The model is a dependency. Production engineering begins when you treat it like one.
Start with a task budget
A useful budget describes more than a maximum bill. It includes how long the user can reasonably wait, how much work one request may consume, and how much uncertainty the feature can tolerate. Those limits should follow the interaction rather than a benchmark leaderboard.
For a draft response, a short wait may be acceptable if the interface clearly shows progress and allows the user to continue reading. A long batch classification job has a different contract. Treating both as synchronous requests because they call the same API would hide an important product distinction.
Write down the deadline before implementing retries. Include time spent retrieving context, waiting in a queue, generating output, validating it, and presenting the result. The model’s response time is only one part of the journey. A fast model behind a congested queue can still produce an unusable experience.
The budget should also describe cancellation. If the user leaves, should the task stop, finish for later, or remain available through a job identifier? None of those choices is universally correct, but an accidental choice is difficult to explain.
Estimate the complete unit of work
The cost of one API call is not necessarily the cost of one successful task. A task may require retrieval, multiple model calls, validation, a retry, and human review. Some inputs may be much longer than others. Failed attempts can still consume resources.
I would estimate a representative task as a sequence, then measure the actual distribution once the feature is running. Separate ordinary work from unusually large inputs and repeated failures. An average alone can hide the small group of requests that dominate cost or latency.
Avoid hardcoding a business model around a price remembered from a launch announcement. Keep provider pricing assumptions dated and replaceable. More importantly, keep usage attributable to an operation or customer without putting confidential content into the accounting record.
For the support draft example, a budget limit might stop automatic attempts and preserve the conversation for manual work. That is a product behavior, not just an infrastructure exception. It belongs in the interface and in the tests.
Make overload an explicit state
Rate limits are not the same as a server crash. They can indicate that the application is asking for more work than a provider currently permits. Retrying every rejected request immediately can turn a temporary constraint into a persistent queue.
OpenAI’s rate-limit guidance describes request and token constraints and recommends retry strategies such as exponential backoff. The application still needs bounded attempts, a total deadline, and a decision about what to do when those bounds are reached. Backoff is a scheduling mechanism, not a guarantee that the task will eventually fit.
At the HTTP layer, RFC 9110 defines Retry-After as either a delay or a date. If an integration uses that information, parse the documented form rather than assuming every value is an integer. Keep the user’s task deadline in view even when the server suggests waiting longer.
A queue should have a capacity policy too. When it is full, the system needs to reject, defer, or prioritize work transparently. Quietly accepting an unlimited backlog makes the interface appear successful while moving the failure into the future.
Caching requires a meaning of sameness
Caching sounds obvious until two apparently identical requests belong to different customers. A response can depend on account permissions, document versions, language, system instructions, retrieval results, or the model configuration. The visible prompt alone may not identify the work safely.
Start with a specific reusable artifact. Perhaps the system can cache a parsed public document or a retrieval result tied to an immutable version. That is easier to reason about than caching a personalized final answer across users.
A cache key should represent the inputs whose changes make the result obsolete. A retention policy should describe when the stored result disappears. Access control must apply to cached data as well as fresh data. Faster retrieval does not make an unauthorized answer acceptable.
It is also reasonable not to cache initially. If reuse is rare or invalidation is unclear, caching adds a second state system before it solves a measured problem. The relevant question is which repeated work is both expensive and safe to reuse.
Minimize the information crossing the boundary
The support conversation may contain information the model does not need. Before sending the entire history, identify the fields required for the draft and the account context needed to interpret them. Reducing irrelevant material can simplify both the privacy boundary and the task itself.
This is an architectural decision rather than a promise that a prompt can enforce. The application controls what it sends, which provider configuration it uses, where diagnostic records go, and who can inspect them. Those settings need verification against the actual service contract and deployment.
Logs deserve particular attention. Recording every prompt for debugging can create another store of customer information with different access and retention behavior. Start with operation identifiers, timings, error categories, and configuration versions. Retain content only for a defined purpose under an appropriate policy.
If a task requires information that cannot cross the chosen boundary, the answer may be a different deployment arrangement or a manual workflow. A more elaborate prompt does not resolve a data-handling constraint.
Evaluate the fallback as a feature
A fallback model is not automatically an equivalent replacement. It may have different output behavior, context limits, or failure modes. The system should evaluate whether the fallback can meet the same acceptance criteria before using it silently.
For the draft response, a simpler fallback could be a structured blank template with the relevant conversation linked beside it. That may be more honest and useful than presenting a weak generated answer as normal output. The user can still complete the task, and the system preserves what has already been collected.
An LLM workflow needs an exit ramp for precisely this reason. Recovery should retain context, explain the next action, and avoid forcing the user to reconstruct the request. A technically successful fallback that loses the work is not much of a recovery.
Test the fallback with real interface states: timeout, exhausted budget, provider rejection, invalid output, and cancellation. Check that retry buttons resume or create work according to a deliberate rule. Otherwise the recovery interface can multiply the very requests that caused the problem.
The fallback is part of the product’s normal operating range, even when you hope it stays quiet.
Operate with evidence
A compact operational view should connect latency, resource use, validation failures, and completed tasks. Provider uptime alone cannot tell you whether users are getting useful drafts. Neither can a count of successful HTTP responses.
Keep a versioned evaluation set for representative tasks and known difficult cases. When prompts, retrieval, or models change, compare outcomes against that set and review material differences. Operational monitoring and task evaluation answer different questions; a stable system can still produce worse results after a change.
The first production design does not need every infrastructure component available. It needs a clear task contract, bounded resource use, observable states, and a way to recover. Add queues, caches, routing, or additional models when they solve a concrete constraint. The boring questions make those choices easier because they tell you what the system is supposed to protect.