What you'll build: a semantic response cache that sits in front of your agent and cuts latency and model spend. An exact-hash fast path, a document-rag near-duplicate match (DataK3 auto-embeds every cached prompt — no manual vectors), and full answers in S3 — all wrapped in a small Ignite function on the request path, feeding an event-sourced savings warehouse that tracks hit-rate, dollars saved, and embedding drift. One bucket.

Why semantic, and why a warehouse?

A large share of production traffic is the same question, reworded — "summarize Q2 revenue" vs. "give me the Q2 earnings in three bullets." An exact-string cache misses all of it; a semantic cache reclaims those calls without a quality hit. And a cache you can't measure is a cache you can't tune: best practice is to log the score of every lookup, so you can watch hit-rate per model over time and catch your embedding model drifting away from your query distribution. That's why the cache is also a warehouse.

What you'll learn:

  • Use document-rag so DataK3 auto-embeds cached prompts — the query embeds server-side on search, no manual vectors.
  • Put the exact → semantic → resolve → write-back → log loop in an Ignite function on the hot path.
  • Turn the cache into a warehouse: hit-rate over time, dollars saved, and embedding-drift alarms.

Problem

You are an agent/app developer paying for the same question over and over — production traffic is full of the same intent, reworded, and every one of those calls burns latency and model spend that an exact-string cache silently misses. This tutorial gives you a semantic cache that reclaims those calls without a quality hit (exact-hash fast path, document-rag near-duplicate match, answers in S3) and, because a cache you can't measure is a cache you can't tune, an event-sourced warehouse that tells you your real hit-rate, the dollars saved per model, and when your embedding model is drifting away from your query distribution — so you can cut inference cost with numbers to prove it, all in one DataK3 bucket.

NOTE

Connect the DODIL MCP once — see the two-minute setup. Each step below shows the CLI and an Ask your agent tab.

How the pillars map

PillarHoldsExample keys / columns
S3 (object store)Full answer payloads + the cached prompt text.cache/answers/9f2c.json, cache/prompts/kimi-k2.6/v3/9f2c.txt
Vector (document-rag)Auto-embedded prompts for near-duplicate lookup.collection prompts_docs
SQL (tables)Entry index + TTL, and an event log for analytics.cache(prompt_hash, …, expires_at), cache_events(…)

One bucket, three pillars over the same rows — no ETL, no second copy, no engine to enable.

Prerequisites

  • A DODIL organization, and the dodil CLI authenticated (dodil auth login) or the DODIL MCP connected to your agent.

Step 1 — Bucket + the semantic substrate (document-rag)

One recipe stands up the vector side. Point document-rag at cache/prompts/ and DataK3 embeds every prompt you cache on write — and data search embeds the incoming query for you, server-side. No manual vectors.

You

Create a DataK3 bucket called agent-cache, then install document-rag with name-prefix prompts scoped to the cache/prompts/ folder so cached prompts are auto-embedded.

DODIL MCP tools called
data_bucket_create→data_recipe_install
Agent

Created agent-cache and installed document-rag (collection prompts_docs + ingest rule prompts_docs_rule, jina-embeddings-v4, cosine) scoped to cache/prompts/. Every prompt written under that prefix now embeds automatically; searches embed the query on the fly.

The recipe provisions a collection named prompts_docs plus an ingest rule that watches cache/prompts/** — anything you write there is chunked, embedded (jina-embeddings-v4), and searchable. There is no vector store to manage: the collection is just the vector pillar of this one bucket.

Step 2 — The warehouse: entries + an event log

Two tables. cache is the entry index (one row per cached prompt, with TTL and running savings). cache_events is append-only — one row per lookup — and it's what makes real analytics possible. Both are keyed, so writes are idempotent upserts: re-runs and shard retries are safe.

You

In agent-cache, create a cache table keyed on prompt_hash (model, template_version, created_at, expires_at, last_used, s3_key as string; hits, tokens_saved as long; cost_saved_usd as double) and a cache_events table keyed on event_id (ts, prompt_hash, model, template_version, outcome as string; similarity, cost_saved_usd as double; tokens_saved, latency_ms as long).

DODIL MCP tools called
data_table_create
Agent

Created cache (pk prompt_hash, 10 columns) and cache_events (append-only, pk event_id, 10 columns). Every lookup writes an event; the entry table carries TTL and running savings.

Step 3 — The cache service (an Ignite function)

A cache is middleware — it runs on every request, in front of the model. Wrap the whole loop in an Ignite function: exact hash first, then a scoped semantic match, resolve from S3, and on a true miss call the model and write back. It scales to zero between bursts. You deploy it in Ship it; here is the logic it runs.

# cache_service.py (essence) — an Ignite function in front of the model
 
from openai import OpenAI
 
# The model this cache fronts — same OpenAI-compatible endpoint + OIDC token as the rest of DODIL.
def dodil_token():
    r = requests.post("https://id.dodil.io/realms/dodil/protocol/openid-connect/token", data={
        "grant_type": "client_credentials",
        "client_id":     os.environ["DODIL_SERVICE_ACCOUNT_ID"],
        "client_secret": os.environ["DODIL_SERVICE_ACCOUNT_SECRET"]})
    return r.json()["access_token"]
 
llm = OpenAI(base_url="https://api.dodil.io/v1", api_key=dodil_token())
 
def handle(req):
    prompt, model, ver = req["prompt"], req["model"], req["template_version"]
    h  = sha256(f"{model}:{ver}:{normalize(prompt)}")     # scope the key by model + version
    t0 = now_ms()
 
    # 1) exact fast path (SQL), TTL-checked in the same query
    row = sql_one(f"SELECT s3_key FROM cache WHERE prompt_hash='{h}' AND expires_at > '{iso_now()}'")
    if row:
        log_event(h, model, ver, "exact_hit", None, req["est_tokens"], now_ms()-t0)
        return s3_get(BUCKET, row["s3_key"])
 
    # 2) semantic near-duplicate — document-rag; `data search` embeds the query server-side.
    #    Keep only hits from THIS model+version, above a high score.
    for hit in data_search(prompt, collection="prompts_docs", top_k=3, min_score=0.92):
        if hit.key.startswith(f"cache/prompts/{model}/{ver}/"):
            log_event(h, model, ver, "semantic_hit", hit.score, req["est_tokens"], now_ms()-t0)
            return s3_get(BUCKET, f"cache/answers/{hit.hash}.json")
 
    # 3) true miss -> call the model, then write back (prompt object auto-embeds)
    ans = llm.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}]).choices[0].message.content
    s3_put(BUCKET, f"cache/answers/{h}.json", ans)
    s3_put(BUCKET, f"cache/prompts/{model}/{ver}/{h}.txt", prompt)   # document-rag embeds this
    upsert_cache(h, model, ver, s3_key=f"cache/answers/{h}.json")
    log_event(h, model, ver, "miss", None, 0, now_ms()-t0)
    return ans

The model call is real — the OpenAI SDK pointed at https://api.dodil.io/v1 with an OIDC service-account token (kimi-k2.6 is the live chat model). The DataK3 reads and writes it wraps are exactly the CLI operations in Steps 4–5.

Step 4 — Inside a lookup (exact → semantic → resolve)

Here's what the function does each call — you can run the same three steps by hand (or ask your agent):

You

In agent-cache, check for an unexpired exact hit on prompt hash 9f2c; on a miss, find a semantically equivalent prompt above score 0.92 in the prompts_docs collection; then give me a link to the stored answer.

DODIL MCP tools called
data_sql→data_search→data_object_url
Agent

No exact hit. Semantic match: 0.94 near-duplicate under cache/prompts/kimi-k2.6/v3/. Answer link (valid 5m): https://object.uk-lon-1.dodil.io/agent-cache/cache/answers/9f2c.json?X-K3-Token=…&X-K3-Expires=… — served from cache, model skipped, ~1830 tokens saved.

Step 5 — Write-back on a true miss

Store the answer, store the prompt (which auto-embeds), upsert the entry, and append the event. The merge-keyed upserts are idempotent — a retried write lands the same row, never a duplicate.

You

In agent-cache, store answer.json and the prompt text for hash 9f2c (model kimi-k2.6, template v3, 7-day TTL), upsert its cache row, and append a miss event.

DODIL MCP tools called
data_object_create→data_table_upsert
Agent

Stored cache/answers/9f2c.json and cache/prompts/kimi-k2.6/v3/9f2c.txt (queued for embedding), upserted the cache row (7-day TTL, wal_written), and appended a miss event. Next identical or near-identical prompt hits.

TIP

A full-row upsert writes the row you pass — include every column, using an explicit null for the ones you don't have yet (the similarity on a miss event, above). Rows whose key column is null or empty are dropped, and each prompt object embeds as one collection entry.

Step 6 — The savings warehouse

Now the event log pays off: hit-rate over time, dollars saved, the embedding-drift alarm, and eviction — all in SQL, in place.

You

In agent-cache, show today's hit-rate and dollars saved per model, the weekly trend of mean semantic-hit score (a drift check), then evict expired entries.

DODIL MCP tools called
data_sql→data_table_delete-rows
Agent

Today: kimi-k2.6 hit-rate 61% (exact 22% + semantic 39%), $84 saved, ~2.1M tokens. Mean semantic-hit score by day: 0.95 → 0.94 → 0.93 — a gentle decline worth watching (re-embed or retune before it drifts under threshold). Evicted 3 expired entries.

IMPORTANT

Threshold and scope are correctness — and security. Serve semantic hits only above a high --min-score (≈0.92+), and scope every match to the same model + template version (the key prefix here) so a hit can't leak from a different setup. Semantic caches can be poisoned — a crafted prompt engineered to collide with a sensitive cached answer — so scope by tenant/identity too, and never let one user's cache serve another's.

DataK3 vs. the three-system stack

ConcernThree-system stackDataK3
Auto-embed prompts + query embedVector DB + a separate embedding service you rundocument-rag embeds on write; data search embeds the query
Big answer payloadsObject store, separate from the indexS3 in the same bucket, one auth
Hit-rate, drift, $ savedRedis counters + a metrics DB + a dashboardAn append-only SQL table + DuckDB queries

Troubleshooting

  • Ingestion lag. A just-written prompt embeds a moment later, so a burst of near-duplicates right after a miss may not hit semantically until it's indexed — the exact-hash path still catches verbatim repeats instantly. Watch dodil data ingest jobs.
  • Threshold = correctness. Too low a --min-score serves wrong answers; prefer false misses to false hits, and tune against your own traffic.
  • Drift. If the mean semantic-hit score slides over weeks, your query distribution has moved past the embedding model — re-embed or swap models before the hit-rate collapses.
  • No native TTL sweeper. Eviction is your delete-rows on expires_at (cron/loop it), followed by vacuum to reclaim storage.

Test

Verify the substrate, the write-back, and the warehouse actually landed. Reads are read-your-writes by default, so a row or object written above is visible immediately — no compaction step.

export BUCKET=agent-cache
 
# 1) The bucket exists
dodil data bucket get "$BUCKET"
# expect: bucket "agent-cache" returned with its metadata
 
# 2) document-rag stood up the auto-embedding collection
dodil data vector collection list --bucket "$BUCKET"
# expect: a collection named prompts_docs (jina-embeddings-v4, cosine) scoped to cache/prompts/
 
# 3) Both warehouse tables exist
dodil data table list --bucket "$BUCKET"
# expect: both `cache` and `cache_events` listed
 
# 4) The write-back row is present and carries a TTL
dodil data sql \
  "SELECT prompt_hash, model, template_version, expires_at, s3_key FROM cache WHERE prompt_hash='9f2c'" \
  --bucket "$BUCKET"
# expect: 1 row — model kimi-k2.6, template_version v3, expires_at set, s3_key cache/answers/9f2c.json
 
# 5) The stored answer object is resolvable
dodil data object url cache/answers/9f2c.json --bucket "$BUCKET" --expires 300
# expect: a pre-signed https URL (object.uk-lon-1.dodil.io/…?X-K3-Token=…&X-K3-Expires=…) valid 5 minutes
 
# 6) The event log recorded the lookup
dodil data sql \
  "SELECT outcome, COUNT(*) AS n FROM cache_events GROUP BY outcome" \
  --bucket "$BUCKET"
# expect: at least one row — outcome 'miss' with n >= 1

Ship it — cache-service on the request path (compile → deploy → scale-to-zero)

The cache_service above is the code; here it becomes the running endpoint your agent calls in front of the model — request-invoked, scale-to-zero, shipped straight from source with the managed build (no external CI or PaaS). (The full git → CI → registry → deploy lifecycle, with versioning and rollback, is validated end to end in Ship a DODIL App; this is the compact form.)

The function reads Models and DataK3 at runtime on its own behalf, so give it a least-privilege service account, then compile and deploy in one command:

You

Create a runtime service account for cache-service with Ignite developer + DataK3 editor roles, then deploy cache_service to Ignite from ./cache_service as a scale-to-zero Python app my agent can call.

DODIL MCP tools called
auth_service-account_create→auth_service-account_grant-role→ignite_app_deploy
Agent

Created service account cache-service (ignite.developer + k3.editor), then deployed cache-service — request-invoked, scale-to-zero, public FQDN on ignite.dodil.cloud. Point your agent's model calls at it: exact + semantic hits return from S3, misses fall through to the model and write back.

Conclusion

A semantic cache — exact SQL, document-rag near-duplicate match, S3 answers — wrapped in an Ignite function on the request path, that reports its own hit-rate, dollars saved, and embedding drift from an event-sourced warehouse. One bucket, one recipe, one function, CLI or agent.

Next steps:

One-shot

Hand this to an agent with the DODIL MCP connected to reproduce the whole tutorial end-to-end:

Build a semantic response cache + savings warehouse in one DataK3 bucket called agent-cache:
 
1. Create the bucket agent-cache, then install the document-rag recipe on it with name-prefix
   prompts scoped to the folder cache/prompts/ so every cached prompt auto-embeds on write
   (data_bucket_create, data_recipe_install).
2. Create two merge-keyed tables (data_table_create): `cache` keyed on prompt_hash with columns
   model, template_version, created_at, expires_at, last_used (string), hits, tokens_saved (long),
   cost_saved_usd (double), s3_key (string); and `cache_events` keyed on event_id with columns
   ts, prompt_hash, model, template_version, outcome (string), similarity, cost_saved_usd (double),
   tokens_saved, latency_ms (long).
3. Deploy the cache_service function to Ignite (ignite_app_deploy) so the agent calls it in front of
   the model: exact-hash SQL lookup, then a document-rag semantic match (data_search over the
   prompts_docs collection) scoped to model+version above min-score 0.92, resolve the answer from S3,
   and on a true miss call the model and write back.
4. Simulate one true-miss write-back (data_object_create for cache/answers/9f2c.json and
   cache/prompts/kimi-k2.6/v3/9f2c.txt, data_table_upsert into cache with a 7-day TTL, and a second
   data_table_upsert of a miss event into cache_events — include every column, explicit nulls).
5. Query the warehouse (data_sql): hit-rate and dollars saved per model, and the weekly trend of
   mean semantic-hit score as a drift check. Then evict expired entries with data_table_delete-rows
   on expires_at and vacuum the cache table.
 
Serve semantic hits only above min-score 0.92 and only from the same model+template version (key
prefix); never let one tenant's cache serve another's.

Sources: Semantic caching solutions 2026 (Maxim) · Semantic cache for LLM inference (Spheron) · Key-collision attack on LLM semantic caching (arXiv)