Semantic Caching
Opt-in, per-workspace caching that serves a repeated or reworded prompt from cache — for free, with no provider call. Off by default; enable it when you want it.
Key Concept
When caching is on, a cache hit returns a previously stored answer for $0 — no model is called — while still reporting the original token counts so your usage data stays consistent. Each workspace's cache is completely private to that workspace.
How It Works
On each eligible request, Antbase checks the workspace cache in two stages before routing to any model:
- Exact match (instant). If the exact same prompt was cached, its stored answer is returned immediately.
- Similarity match. Otherwise Antbase embeds the prompt and compares it against recently cached prompts for that workspace. If the closest one is at or above the similarity threshold, that answer is returned.
- Miss. If neither stage hits, the request is routed to a model as normal, and the fresh answer is stored for next time.
Free on a Cache Hit
A cache hit costs nothing. There is no upstream provider call, so you are charged $0 for that request. The response carries the original token counts from when the answer was first generated, so downstream accounting and analytics behave as expected.
Privacy & Isolation
The cache is per-workspace and private. A workspace's cached prompts and answers are never shared with, or visible to, any other workspace. A hit can only ever come from your own workspace's prior requests.
Threshold & TTL
| Setting | What it does | Default |
|---|---|---|
| Similarity threshold | Minimum similarity (0–1) for a reworded prompt to count as a hit. Higher = stricter, fewer but safer hits. | 0.92 |
| TTL | How long each cached entry stays eligible before it expires and is ignored. | Per entry |
Raise the threshold when you need answers to match the prompt very precisely; lower it to catch looser paraphrases. Both the threshold and TTL are configurable per workspace.
What Is Not Cached
Only plain, single text completions are cacheable. The following are never cached — they always route to a model:
- Requests that use tool calls / function calling
- Requests with image inputs (vision)
- Requests asking for multiple completions (
n > 1)
Non-streaming only
Semantic caching currently applies to non-streaming requests only. Streaming requests (stream: true) always route to a model and are not served from cache.
Enabling Semantic Caching
Caching is off by default. Turn it on per workspace at /app/settings, where you can also adjust the similarity threshold and entry TTL. No code changes are required — once enabled, eligible requests are served from cache automatically.
Example: Second Call Served From Cache
With caching enabled, an identical — or paraphrased — second call returns the stored answer for free.
Python (OpenAI SDK)
from openai import OpenAI
client = OpenAI(
base_url="https://antbase.ai/v1",
api_key="ant_your-api-key",
)
# 1) First call — cache MISS. Routed to a model, billed normally.
first = client.chat.completions.create(
model="ant:auto",
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(first.choices[0].message.content) # "Paris."
# 2) Identical prompt — exact cache HIT. Returned instantly for $0.
again = client.chat.completions.create(
model="ant:auto",
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
# 3) Paraphrased prompt — similarity cache HIT (>= 0.92). Still $0.
paraphrase = client.chat.completions.create(
model="ant:auto",
messages=[{"role": "user", "content": "France's capital city is which one?"}],
)
# Cache hits return the SAME stored answer and the ORIGINAL token counts,
# but cost nothing — no provider call was made.
print(again.choices[0].message.content) # "Paris."
print(paraphrase.choices[0].message.content) # "Paris."Because the cache is private to your workspace, these hits can only ever come from your own prior requests.