Why AI Features Break on Deadline Day, and How to Keep Them Fast

A deadline-day outage can begin with a connection pool filling up while the CPU still looks healthy. Duplicate background workers, undisposed HTTP connections, nested retries and an ineffective cache can all increase the number of connections a service needs.

Under light traffic, those faults can stay hidden. A load test needs to hold traffic long enough for connections and queues to accumulate.

Enterprise Peaks Come From Shared Deadlines

Consumer apps get traffic spikes from news and marketing. Enterprise systems get them from calendars. Month-end close, a quarterly report, a survey window that closes on Friday at five: hundreds of people who ignored the system for weeks all open it in the same afternoon.

Capacity needs a measured workload. A test with 100 people reading cached reports says little about 100 people uploading files and asking an AI assistant to analyse them. Record the mix, ramp up gradually and hold the expected peak long enough to expose leaks and growing queues.

Count requests in flight

Compare a database query lasting 20 milliseconds with an illustrative model call lasting 20 seconds. A streamed response keeps a connection open while generation continues. For that example, if 100 requests arrive each second and each one stays open for 20 seconds, 2,000 of them are in flight at once.

That changes several things that used to be safe defaults. Connection pools sized for fast queries run dry. Web servers that allow a few hundred concurrent requests start queueing. Model providers enforce limits on tokens and requests per minute, usually shared across a whole account, so one busy feature can starve every other feature that uses the same key.

The designs that hold up treat AI calls as slow, scarce work. Each feature gets its own concurrency limit and, where possible, its own quota. Long jobs go into a queue with a visible position, and streamed answers can show output before generation finishes. Every call has a timeout and can be cancelled when the user leaves the page.

Avoid repeated work

Efficiency in AI systems starts with the same move as efficiency in databases: avoid repeating work. Some providers discount cached input when requests meet their prefix, length and retention requirements, so an assistant that resends the same long instructions and reference documents on every turn should be structured to reuse them. Results that do not change, such as a classification of last quarter's documents, belong in a cache with a clear expiry.

Model choice is the second lever. Sorting a ticket, extracting fields from an invoice or checking whether a sentence mentions a supplier is work for a small, fast model. Reasoning across a contract or drafting an analysis justifies a larger one. Use tested code for arithmetic, with explicit units, rounding and decimal precision.

Cache hits need correctness checks as well as a counter. Include the user’s permissions and the source version in a cache key, and invalidate results when either changes. A high hit rate is a failure if employees receive outdated answers or another tenant’s records.

Retries Are Load Too

Retries at several layers multiply. If an upload client, gateway and storage adapter each make three attempts, one operation can generate 27 downstream attempts. Give one layer responsibility for retries and keep an overall attempt budget.

For AI systems this matters twice over, because an agent that retries a tool call can repeat an action. Retries need exponential backoff with some randomness and an overall attempt limit. Calls that change records should carry an idempotency key where the receiving service supports one. The receiving service must enforce deduplication. Where an email or payment API lacks that support, reconcile its status before trying again.

Measure the Slowest Requests

An average hides the requests people complain about. Measure p95 and p99 latency, error rate, queue age and completed work per minute under the same workload. Report cancelled and rejected requests too; excluding them can make an overloaded service look fast.

For AI features, measure time to the first useful output separately from time to completion. A progress message can explain a queue, while streaming helps once generation starts. Set targets from the task and user testing rather than assuming everyone tolerates the same delay.

Before the next deadline, test the expected peak and an agreed safety margin with AI features switched on. Watch open connections, queue age and rate-limit responses alongside CPU. The Juejin AI-native tutorial offers an application-stack overview; capacity planning still needs measurements from the workload you will actually run.