Using the GPT API in Production: Rate Limits, Costs, Fallbacks and the Things the Quickstart Skips

The GPT API quickstart gets a call working in five minutes. Running it in production, where it handles real traffic, real cost and real failures, requires things the quickstart never mentions. This is the list we work through on every deployment.

Rate limits

Every tier has requests-per-minute and tokens-per-minute ceilings, and traffic is bursty. Size client-side concurrency below the ceiling; back off exponentially with jitter on a 429; queue requests so a burst is smoothed rather than dropped; and watch the rate-limit headers so pressure is visible before it becomes errors. Anything not latency-sensitive, nightly classification, bulk summarisation, goes through the batch endpoint at a fraction of the cost and none of the contention.

Cost

The largest lever is routing. Most requests in a real system are simple, classify, extract, decide whether to escalate, and a small model handles them at a fraction of the price. Reserve the large model for steps that need reasoning. That alone typically halves spend with no measurable quality drop. Add prompt caching for repeated system context, tight output token limits, and a per-feature budget with alerts. Model the monthly cost at expected volume before launch; the first invoice is the wrong time to learn.

Fallbacks

Providers have outages. Models get updated underneath you, sometimes changing behaviour without a version bump you noticed. A production system needs a thin abstraction so a request can route to a second provider, Claude, an open-weight model, another endpoint, when the primary is down, rate-limited or behaving differently. Our OpenAI vs Claude comparison covers the practical differences when you set this up.

Timeouts and partial failures

Long generations stall. Set timeouts, stream where the interface allows it, and decide in advance what the user sees when a call fails: a retry, a fallback answer, or an honest error. Never let a hung request hold a connection indefinitely.

Structured outputs

If the output feeds another system, constrain it: JSON schemas and function-calling formats the model must satisfy, validated on receipt, with a retry on schema failure. Free-text output parsed with regex is the quickstart's habit and production's bug.

Evaluation and logging

An evaluation set of real cases, scored before launch and on every prompt or model change. Retrieval quality measured separately where the system retrieves first, our knowledge base methodology explains why. And log every call: prompt, parameters, output, tokens, latency, cost. When quality drops, this is how you find out whether the model changed, the prompt regressed or the inputs shifted.

Security and data

Never send data the user is not entitled to see. Redact or tokenise personal data where the use case allows. Know which provider settings govern training on your data, and where the data is processed: for clients under GDPR, PDPL or HIPAA that decides whether the API can be used at all, or whether an open-weight model in your own environment is required.

An incident, and what it taught us

A client's support assistant went quiet on a Tuesday afternoon. Not down, exactly: requests were accepted and then hung for forty seconds before failing. The provider was having a partial outage that never made it to their status page. The assistant had no timeout, so every request tied up a worker until the platform killed it, and within minutes the whole application was unresponsive, including the parts that had nothing to do with AI. The fix took an afternoon: a fifteen-second timeout, a circuit breaker that stops sending requests to a provider after a run of failures, and a fallback route to a second provider. The lesson was that the model is a network dependency like any other, and the discipline that applies to payment gateways and email services applies to it. Most teams learn this the same way we did.

Routing rules that hold up

The routing that halves cost is not complicated, but it needs to be explicit rather than left to whoever wrote the first prompt. We classify each request type once and assign it a model. Intent detection, language detection, simple extraction and yes-or-no decisions go to the smallest model that passes the evaluation set for that task. Multi-step reasoning, long document synthesis and anything customer-facing that needs judgement go to a frontier model. Then two rules on top: any request that fails validation on the small model is retried once on the large one, and any request over a length threshold skips the small model entirely. The routing table lives in configuration, not code, so it can change when a new model ships. And the evaluation set is run per route, because a small model that passed six months ago may not pass on the requests you receive today.

Prompt versions and the changes nobody announces

Model providers update versions, sometimes with notice and sometimes with a note in a changelog nobody reads. Prompts that worked for months start producing slightly different output, and the first sign is a support ticket. Two habits prevent this from becoming a crisis. Pin to a dated model version rather than the alias that floats to the latest, so updates happen when you choose. And keep prompts in version control with the evaluation scores attached, so a change to either can be reviewed like code. When a new model version arrives, run the evaluation set against it in a branch, compare, and switch when the numbers say so. It is unglamorous and it is the reason our production systems do not break on the provider's release schedule.

What the logs should let you answer

Six questions, and if the logs cannot answer them the logging is not finished. Which prompt version produced this output? Which model, and which version of it? How many tokens in and out, and what did the call cost? How long did it take, end to end and at the provider? Did validation pass, and if not, what was retried? And which user or session was this, so that a complaint can be traced to the exact exchange. With those, a drop in quality can be diagnosed in an hour: a model change, a prompt regression, a shift in the inputs, or a retrieval failure upstream. Without them, the diagnosis is a meeting, and the meeting usually concludes that the model is unreliable, which is almost never the actual cause.

A checklist to hand to your team

Before a language model call goes to production: a timeout, a retry policy with backoff and a cap, a circuit breaker per provider, a second provider or model wired in and tested by actually failing the first, a token budget per request and per user, output validation against a schema, a dated model version rather than a floating alias, prompts in version control with scores, structured logs with the six fields above, and a monthly review of cost per request type. It is a short list. Most of it is an afternoon each. The teams that skip it are not saving time; they are deferring it to an evening when the provider is down and the customers are not.

When to bring in help

If your team has shipped web services before, most of this article is applying habits you already have to a new dependency, and you can do it yourselves. Bring in help when the model calls are on a customer-facing path and there is no evaluation set, when the monthly bill is growing faster than usage, or when an outage has already happened and the post-mortem concluded that the model was unreliable. Those are the three situations we get called for, and in each the fix is the plumbing in this article rather than a different model.

What we do

Our LLM integration practice builds exactly this layer, routing, fallbacks, structured outputs, evaluation, logging, underneath the AI applications and agents we deliver, because the model is the easy part. If you are still sorting which of your tasks a generative model should do at all, our guide to generative AI tasks is the place to start.