How We Reduced Agent Latency with Semantic Caching

We cut a large share of our agent's response latency by adding semantic caching — reusing a prior answer when a new question means roughly the same thing, matched by meaning rather than exact text. Here's how we approached it, what we measured, and the tradeoffs we had to navigate to keep answers correct.

The latency we were fighting

A grounded answer isn't free. Each one means embedding the question, retrieving relevant chunks, and a model call to generate the response. The model call dominates — it's where the seconds go. On a busy site, the same questions recur constantly: "how do I install this," "what are your limits," "do you support X." Regenerating an identical-in-meaning answer every time is wasted latency and wasted spend. That repetition was our opportunity. See what is semantic caching for the concept.

Why exact-match caching was useless

Our first instinct, exact-string caching, barely helped. Visitors phrase the same question a dozen ways; "I forgot my password" and "how do I reset my login" share almost no characters. An exact cache treated them as different and almost never hit. We needed to match on meaning, not text.

The semantic approach

We already had embeddings in the pipeline for retrieval, so the building block was there. When a question arrives, we embed it and compare that vector to embeddings of recently answered questions. If a past question is similar enough — above a cosine-similarity threshold — we serve its cached answer and skip the model call entirely. A hit returns in a fraction of the time of a fresh generation, and costs nothing in model spend.

The threshold problem

The hard part was the threshold. Too loose, and the cache serves a subtly wrong answer because it treated two genuinely different questions as the same — the worst failure mode, since it's confidently incorrect. Too tight, and the cache rarely hits and the feature does nothing. We started conservative, deliberately accepting fewer hits to guarantee correctness, then measured real traffic to see how much headroom we had before mismatches appeared. Accuracy won every time it conflicted with hit rate.

Keeping it fresh

A cache that serves stale answers is worse than no cache. When a site's content changes and we re-crawl, answers tied to the old content must be invalidated, or the agent confidently repeats outdated information. We tied cache eviction to the re-crawl, so updating content clears the answers derived from it. Freshness discipline is what makes the speed safe.

What we bypass

Not everything is cacheable. Anything personalized — a question about this customer's order — or anything that calls a tool must bypass the cache and run live, because the answer depends on state the cache can't hold. We cache general, content-grounded answers and route everything else around it. Drawing that line clearly is essential.

The result

The payoff is a meaningfully faster agent for the common questions that dominate real traffic, with lower model spend, and no loss of accuracy because the threshold stays conservative and the cache evicts on change. It's one of the higher-leverage performance wins we shipped.

Getting started

Our grounded agents handle this for you. Start on the AIML.chat free plan, compare tiers on the pricing page, and read semantic caching explained for the fundamentals.