v1.0.0

Caching, fallback, and failure containment

Every optimization has a cache-invalidation and correctness story.

Estimated study time: 50 minutes

Progress is stored on this device.

Learning outcomes

  • Compare exact-response, retrieval, and prompt-prefix caching
  • Design cache keys that preserve tenant isolation and document versions
  • Choose safe fallback behavior without silently degrading correctness

Cache the right object

Three distinct caches appear in LLM applications:

  1. Provider prompt-prefix cache: reuse an eligible identical prefix according to the provider’s rules; may reduce billed input cost and/or latency.
  2. Retrieval cache: reuse query-to-document results for an unchanged index and authorization scope.
  3. Final-answer cache: reuse completed answers for identical user-visible requests under carefully defined validity conditions.

Do not mix these up. A prompt-prefix cache does not mean your app has a valid final answer cached.

Cache-key correctness

A dangerous final-answer key is simply hash(question). Two tenants can ask the same question but have different policies. At minimum, consider:

text
hash(tenant_id | normalized_question | doc_index_version
     | prompt_version | model_config | authorization_scope)

The exact schema depends on your system. The authorization scope must be handled carefully; sharing cached answers across permissions can leak information. Invalidate on policy changes. Avoid storing sensitive content in a cache without appropriate encryption, TTL, access control, and deletion mechanisms.

Fallbacks can be unsafe

A common fallback is: “If retrieval fails, ask the model to answer from general knowledge.” For a company policy assistant, this is usually not a safe fallback because the general answer may contradict the actual policy. A better fallback is an explicit abstention or a link to support.

If a model provider is unavailable, consider whether a tested alternative provider can preserve the same privacy, authorization, output schema, and quality requirements. A provider switch is a product change, not a silent implementation detail.

Failure containment

Retries can amplify failures. Combine short bounded retries with backpressure, concurrency limits, and explicit overload behavior. Where relevant, use circuit breakers to stop hammering a failing dependency. A circuit breaker needs a reopening strategy and is not a replacement for monitoring.

Drill: stale answer

At 09:00 an answer is cached with a 30-day refund policy. At 09:15 the policy changes to 14 days. At 09:20 the user asks the same question. If the cache key ignores document version and invalidation, the system serves stale guidance. TTL helps reduce the duration but does not solve an immediate high-risk policy update. Versioned invalidation is more robust.

Hands-on lab

Design a cache key for DocDesk and write three tests: cross-tenant isolation, policy version invalidation, and fallback abstention when retrieval is unavailable. Explain why the question text alone is never a sufficient cache key.

Download the lab kit

Knowledge check

What is wrong with cache key = hash(question)?
What is the safest fallback when no policy evidence is available?

References