Skip to content

Prompt caching ​

Every question an assistant asks a model carries much more than the question: the rules of the assistant, the actions it may run, the conversation so far. Most of it is the same as in the request before. A service with a prompt cache keeps the beginning of a request for a few minutes, and when the next request starts with exactly the same text it reads that part from the cache: cheaper, and quicker to start answering on a long prompt.

HelpChatAI lays out its requests so that the part that repeats comes first. This page says how, what it saves, and what you can do in your own application to keep it working. All three model providers send their requests this way; claude also tells the service where the reusable part ends, while openai and gemini leave it to the automatic cache of their service.

How the cache of Claude works ​

Price of the input
Read at full price1x
Written to the cache, the first time1.25x
Read from the cache0.1x (0.05x on Sonnet 5.5 and Opus 5.5)
  • It is a prefix. The service compares the request with what it keeps from the start, byte by byte: one different character and everything after it is read at full price again.
  • The order is fixed by the API: the tools first, then the system prompt, then the messages.
  • Breakpoints say where the reusable part ends: at most four per request, a cache_control mark on a block. The service writes the text up to each mark and, at the next request, finds the longest one that is still the same.
  • An entry lives five minutes, and every read renews it: a user who asks again within five minutes always finds it.
  • Below a minimum nothing is cached, without an error: 512 tokens on the current models (Haiku, Sonnet and Opus 5.5), 4096 on Haiku 4.5.
  • The cached part is read faster, too: the answer starts sooner on a long prompt. On the short prompts of a help assistant the difference is not measurable.

What HelpChatAI sends, in order ​

A request of the claude provider is laid out from what never changes to what changes with every question:

#PartChangesBreakpoint
1The tools: the actions of the form, the shared ones, the skillswhen the actions registered changeon the last tool
2The system prompt: who the assistant is, its field, the rules, its Instructions, the language, the actions rulenever, for one assistantat its end
3The conversation so far, and the question being askedgrows at the endon the question
4The context of the question: now, the screen, the documentation excerptsevery question-
5In the loop of actions: the calls of the model and their resultsevery roundautomatic, at the end

The fourth part is the one that makes the rest reusable. Now (NowRule, "Now it is Monday, 16 November 2026, 17:47 (UTC+01:00).") changes every minute, the screen every time the user moves, the excerpts with every question: in the system prompt they would make everything after the tools new at every question. So they travel with the question, in a message of their own right after it, and they never go back into the conversation: when the user asks again, the previous question is sent as it was typed, exactly as the cache wrote it, and the whole conversation up to the new question is read from the cache.

The context is a system message, the channel of the application: the model gives it the weight it gives the system prompt, and nothing the user types can pass for it. A model that does not accept a system message among the others (Haiku 4.5) refuses the request; the provider then sends the context as a second text block of the question, for that request and for every later one of the same model in the process. Nothing else changes.

OpenAI, Gemini and the local servers ​

The same two parts, the same order: the system prompt that does not change, the conversation, the question, and its context. Only the place of the context depends on what each API accepts:

ProviderWhere the context of the question goes
openai, Responses API (OpenAI itself)A developer message right after the question: the channel of the application, as system is for Claude
openai, Chat Completions (Azure, Groq, OpenRouter, Ollama, LM Studio...)In the message of the question, in front of it. A second system message is refused by the chat template of several local models, and with the excerpts after the question qwen3-coder lost the request it was given
geminiA second part of the question: this API has no system message among the others

Their caches need no mark, and start on their own above a size of their own:

  • OpenAI caches from 1,024 tokens of identical start, and reports what a request wrote and what it read. Two identical requests of 3,148 tokens, one after the other: the first wrote 3,145, the second read them.
  • Gemini reads the start of a request again on longer prompts. The same request of about 6,000 tokens sent three times: Gemini 3.5 Flash Lite read 2,035 tokens of it from the second time on, Gemini 4 Argon 4,052, Gemini Pro nothing.
  • A local server (Ollama) keeps the start of the last prompts in memory and reports what it reused: it costs nothing either way, but the answer starts sooner. On the questions of the benchmark qwen3-coder:30b reused 15,309 tokens, gemma4:12b 3,770.

So a help assistant with two or three actions sits below these thresholds on OpenAI and Gemini - the questions of the benchmark, about 2,000 tokens each, read nothing there - while one with the skills (about 2,000 tokens of tools on their own) and a conversation of a few questions goes over them. In the conversation suite gpt-6-luna started reading at the seventh question (about 2,200 tokens), gpt-6.1-sol at the sixth, Gemini 4 Argon and Carbon about 600 at the last two, Gemini 3.5 Flash Lite and Pro nothing; qwen3-coder on Ollama from the first.

The settings ​

SettingDefaultWhat it does
Claude.PromptCachingtruePuts the breakpoints. false for a proxy or a compatible service that refuses cache_control: the request keeps the same layout, without marks
NowRule"Now it is %s."The sentence with now, in the context of the question
IdentityRule"Your name is %0:s."The first sentence of the system prompt

An IdentityRule written before 1.12 may still say now (%1:s): it gets now as before and NowRule is not added, but then the system prompt changes every minute and the cache can keep only the tools. The configuration check and /check say so.

OnBuildSystemPrompt has the last word on the system prompt - on the part that does not change. Whatever it adds should not change from one question to the next either: a value of the moment written there (the user logged in, a counter, the time) costs the cache of everything after it at every question.

What it saves ​

Measured with Test\LLMBench on the eight questions of its suite, with claude-haiku-5-5, the default model: once with the cache off (--nocache), once with only the tools and the loop of actions marked (the first version), once as described above.

Input at full priceWrittenReadInput as billed
No cache30,0920030,092
Tools and loop of actions3619,66810,14625,636 (-15%)
Stable system prompt, context with the question2015,36614,73420,701 (-31%)

"Input as billed": input + 1.25 x written + 0.1 x read, in tokens at the full price.

That suite opens a new conversation for every question, so it measures the tools and the system prompt read again (about 1,100 tokens a question) and the rounds of the actions (a question that ran one read about 4,000). The conversation suite (--suite=conversation) asks the same eight questions in one conversation, as a user does, the history never cleared:

claude-haiku-5-5, one conversationInput at full priceWrittenReadInput as billed
No cache36,7570036,757
With the cache2015,88222,16622,089 (-40%)

What each question read grew with the conversation: 1,114 tokens at the first (the tools and the system prompt), then about 200-300 more at every question - the question and the answer before it - up to 6,592 at the eighth, which ran an action. What is never read is the context of each question, about 1,900 tokens of excerpts: it is new every time, and it does not go back into the history, by design. The answers stayed at 100%.

The answers did not change. Citations, facts and actions stayed at 100% for every Claude, OpenAI and Gemini model of the benchmark, and the local models answered as before. On the expertise suite - twelve questions graded by a judge model - moving the excerpts out of the system prompt changed nothing beyond the noise of a judged run (the same model, asked twice, passes two questions more or less):

Passed out of 12Without Expertise, before -> afterWith Expertise, before -> after
Haiku 5.53 -> 45 -> 5
Opus 5.54 -> 77 -> 7
gpt-6.1-sol2 -> 27 -> 8
Gemini Pro3 -> 24 -> 4

None invented a feature of the application after the change; Haiku and Opus had invented one each before.

Measuring it in your application ​

The service reports what it read and wrote with every answer - Claude, OpenAI and Gemini, and a local server that sends the counts. The provider passes them to HelpChatOnTokenUsage (HelpChatAI.Provider), a hook for a log of the costs, called once per request - per round of the loop of actions - on the worker thread that made it:

pascal
uses
  HelpChatAI.Provider;

initialization
  HelpChatOnTokenUsage :=
    procedure(const AUsage: THelpChatTokenUsage)
    begin
      // A worker thread: quick, thread-safe, no user interface here.
      TThread.Queue(nil,
        procedure
        begin
          Log(Format('%s: %d in, %d written, %d read, %d out', [AUsage.Model,
            AUsage.InputTokens, AUsage.CacheWriteTokens, AUsage.CacheReadTokens,
            AUsage.OutputTokens]));
        end);
    end;

The three inputs add up to the whole prompt (Gemini reports no writes, OpenAI and Claude do). A healthy conversation reads more at every question and writes little: what the last answer and the new question added.

Test\LLMBench prints the same counts for every question and adds two tables to its report: Tokens, per target, and Tokens per question, what each question read out of its whole input. --nocache runs the same questions without the cache, to compare; --suite=conversation asks them in one conversation.

What makes it miss ​

The cache is silent: a request that misses still succeeds, it just costs more. What breaks the prefix, from the most common:

  • More than five minutes between two questions. The next one writes again.
  • Actions that change between questions. The tools come first: registering or removing an action, or naming another skill, makes the next request new from the start. Register them once, when the form is created.
  • Another model. Each model has its own cache: switching in the panel starts again.
  • Attachments. They travel with the last question, so when a new question is asked they move and the conversation from the first one on is written again.
  • OnBuildSystemPrompt writing something that changes, as above.
  • A prompt below the minimum of the model: nothing is written, nothing is read.

With a remote engine ​

The server runs the provider, so the cache works there the same way, for each profile: the tools, the system prompt and the conversation of each session. The tools are the actions the client declared: they are cached as long as the declaration does not change.

Released under the Apache License 2.0