.jpg){kind=link}
When context windows were small, everyone worked hard to decide what deserved a place in the prompt. Now that they are large, the habit has quietly disappeared, and a common architecture has become “put in everything that might be relevant and let the model sort it out”.
That works. It is also expensive in three separate ways, and only one of them shows up on an invoice.
The money
Cost scales with tokens, and input tokens are usually the majority of them. Sending fifty thousand tokens of context to answer a question that needed five hundred is a choice you make on every single request.
At low volume nobody notices. At scale it becomes the dominant line item, and by then the padding is load bearing because nobody remembers which parts were actually necessary.
The latency
More input means more time before the first token appears. Prompt processing is not free, and users experience it as the feature being slow rather than as the prompt being large.
This is the cost people misattribute most often. Teams switch to a faster model to fix a latency problem that was caused by their own context size.
The quality
The one that surprises people. More context does not reliably mean better answers, and past a point it means worse ones.
Relevant information sitting in the middle of a long prompt gets used less reliably than the same information near the beginning or the end. Padding a prompt with loosely related material also gives the model more opportunities to build a plausible answer out of the wrong paragraph.
I have watched a system’s accuracy improve by cutting retrieved chunks from twenty to five. Not because five is a magic number, but because fifteen of the twenty were noise, and noise is not neutral.
What to do instead
Retrieve less, but rank it properly. A reranking step that picks the best four chunks out of forty candidates beats sending all forty. The work moves from the expensive model to a cheap one, which is the right direction.
Put the important material at the edges. Instructions at the start, retrieved evidence near the end, close to the question. Position matters more than most people assume.
Summarise history rather than replaying it. A conversation does not need every previous turn verbatim. A running summary plus the last few exchanges holds up well and stops the prompt growing without limit.
Cache the stable part. If a large block of context repeats across requests, prompt caching turns it from a recurring cost into an occasional one. This is often the single largest saving available and it requires no change to behaviour, only to how the prompt is ordered.
Measure tokens per request like you measure latency. If it is not on a dashboard it will grow, because every individual addition to a prompt seems harmless.
The principle
A large context window is a capability, not an instruction. It means you can send a lot when you need to, not that you should send a lot by default.
The discipline of deciding what earns its place in the prompt did not stop being valuable when the limit went up. It just stopped being enforced.