Building an AI Research Agent on DODIL

What you'll build: a persistent research memory bank on DataK3 that answers real questions over ArXiv papers — filtering by metadata, ranking by meaning, and citing the exact source PDF.

What you'll learn:

• How a VECTOR column (or one recipe install) gives a bucket semantic search — no separate vector DB.
• The narrow → rank → resolve retrieval loop that powers a research agent.
• How to run every step two ways — the dodil CLI, or by prompting an agent through the DODIL MCP.

This is DataK3's reference workload. There's a working implementation — davidmiheev/research-agent ("Atlas") — a small Python agent on Dodil Ignite that keeps all of its long-term memory in DataK3 with semantic vector search. This tutorial generalizes the pattern behind it.

Problem

The end user here is a research engineer (or the agent working on their behalf) who needs to answer precise questions over a growing pile of ArXiv PDFs — "what did 2026 cs.CL papers say about expanding transformer context windows?" — and cite the exact source. The naive stack bolts an object store to a separate vector DB to a separate warehouse, then hand-writes CDC glue to keep the PDF, its embeddings, and its metadata in sync; embeddings silently drift the moment a paper is replaced, and "filter by category then search semantics" becomes a cross-system join in app code. The payoff of doing it on one DataK3 bucket: the embeddings live in a VECTOR column right next to the metadata (a merge-keyed table) and the raw PDF (an object), and one question flows narrow → rank → resolve without any inter-system plumbing — so a research agent gets durable, citable long-term memory instead of a brittle three-system pipeline.

Prerequisites

• A DODIL organization, and either the dodil CLI authenticated (dodil auth login) or the DODIL MCP connected to your agent.
• A sample paper to ingest — e.g. 2603.07890.pdf.

Nothing else to turn on: a new bucket has its SQL/vector tables engine ready by default (there is no engine to enable), and it's created with an internal object source already wired for ingestion.

Scenario

A user asks for recent breakthroughs in "transformer context windows." A capable agent should:

1. Use SQL to filter papers categorized cs.CL and published in 2026.
2. Use Vector search to locate the paragraphs that actually discuss context-window expansion.
3. Pull the source PDF from S3 to cite exact findings.

That single question touches structured filtering, fuzzy semantic retrieval, and heavy-payload fetch — the three things DataK3 unifies over one copy of the rows.

How the three pillars map

The agent's loop becomes: narrow with SQL → rank with Vector → resolve with S3.

Step 1 — Create the bucket

export BUCKET=blog-research
dodil data bucket create "$BUCKET" --description "ArXiv research memory bank"

Step 2 — Give the bucket semantic search

A vector isn't a separate store any more — it's a VECTOR(<dim>) column on a table, right beside your other columns. We'll keep chunked paper text in a paper_chunks table whose embedding column holds 2048-dim jina-embeddings-v4 vectors. That's the path this tutorial searches, and the one you control end-to-end.

dodil data table create paper_chunks -b "$BUCKET" \
  --columns-json '[
    {"name":"chunk_id","type":"string"},
    {"name":"arxiv_id","type":"string"},
    {"name":"body","type":"string"},
    {"name":"embedding","type":"VECTOR(2048)"}
  ]' \
  --merge-key chunk_id

# One command: embed collection + ingest rule, in the right order (no vector store to provision)
dodil data recipe install document-rag -b "$BUCKET" --name-prefix papers --folder papers/
 
# Preview exactly what it provisions first (optional)
dodil data recipe show document-rag

Step 3 — Create the metadata table

Create a papers table keyed on arxiv_id, so re-inserting the same paper upserts instead of duplicating. A PRIMARY KEY (merge-key) is required — writes are keyed, so re-runs and shard retries upsert idempotently.

dodil data table create papers -b "$BUCKET" \
  --columns-json '[
    {"name":"arxiv_id","type":"string"},
    {"name":"title","type":"string"},
    {"name":"primary_category","type":"string"},
    {"name":"published","type":"string"},
    {"name":"s3_key","type":"string"}
  ]' \
  --merge-key arxiv_id

Step 4 — Ingest a paper

Upload the PDF to the object store, embed a chunk of its text with jina-embeddings-v4 into paper_chunks, and record the metadata row so SQL can filter it. Three writes, one bucket — no CDC glue between them.

# (a) raw PDF → object store
dodil data object create ./2603.07890.pdf -b "$BUCKET" --key papers/2026/2603.07890.pdf
 
# (b) embed a chunk with jina-embeddings-v4 (2048-dim) and upsert it as a VECTOR literal
VEC=$(dodil ignite models embed jina-embeddings-v4 \
  --input "Streaming attention keeps a bounded KV cache and evicts stale keys, letting the model process million-token contexts with constant memory." \
  -o json | python3 -c 'import sys,json;print("["+",".join(f"{x:.6f}" for x in json.load(sys.stdin)["data"]["data"][0]["embedding"])+"]")')
 
dodil data table upsert paper_chunks -b "$BUCKET" \
  --row "{\"chunk_id\":\"c2\",\"arxiv_id\":\"2603.07890\",\"body\":\"Streaming attention keeps a bounded KV cache...\",\"embedding\":\"$VEC\"}"
 
# (c) metadata row → SQL can filter it
dodil data table upsert papers -b "$BUCKET" \
  --row '{"arxiv_id":"2603.07890","title":"Streaming Attention for Million-Token Contexts","primary_category":"cs.CL","published":"2026-03-22","s3_key":"papers/2026/2603.07890.pdf"}'

Step 5 — Answer the question (narrow → rank → resolve)

SQL narrows the candidates, Vector ranks by meaning, and S3 resolves the exact citation. data vsearch embeds the query text with the same model (jina-embeddings-v4) and returns a cosine distance score — lower is closer.

# (a) SQL narrows the search space
dodil data sql -b "$BUCKET" \
  "SELECT arxiv_id, title, s3_key FROM papers
   WHERE primary_category = 'cs.CL' AND published >= '2026-01-01'
   ORDER BY published"
 
# (b) Vector ranks by meaning (query embedded with the same jina-embeddings-v4 model)
dodil data vsearch -b "$BUCKET" -t paper_chunks --column embedding \
  --text "expanding transformer context window length" \
  --model jina-embeddings-v4 --metric cosine --top-k 4
 
# (c) S3 resolves the exact citation — a pre-signed URL for the cited PDF
dodil data object url papers/2026/2603.07890.pdf -b "$BUCKET" --expires 3600

Query it — one bucket, drop-in clients

The same rows answer by content (SQL) and by meaning (vector) with no ETL and no second copy — and any Postgres/pgvector client points straight at the bucket. data connect prints the endpoints (DB name = bucket, credential = your login token), so psql or a notebook runs the exact same SELECT … ORDER BY embedding <=> '[…]' KNN.

dodil data connect "$BUCKET" -o psql   # postgresql://…@pg.uk-lon-1.dodil.io:5432/blog-research

DataK3 vs. the three-system stack

Troubleshooting

• A recipe auto-embed job failed. If you took the recipe path, poll dodil data ingest jobs -b "$BUCKET" -o json and re-trigger with dodil data ingest trigger --retry-failed. Treat a failed job as needing an explicit retry.
• A KNN over a big table is slow the first time. The default request timeout is 30s; give a cold, large paper_chunks room with --timeout (e.g. --timeout 120s). Once warm, queries return in well under a second.
• Metadata-filtered vector search isn't a one-liner on the CLI. Run the SQL query first and keep the returned arxiv_ids, fuse it into one data sql … ORDER BY embedding <=> '[…]' statement, or just prompt an agent (Step 5).
• Embeddings look stale after a model change. A VECTOR column is pinned to whatever model produced its vectors: switching models means re-embedding the column (recompute + upsert) so query and stored vectors share a space. Don't mix jina-embeddings-v4 vectors with another model's.

Test

Verified end-to-end against a live blog-research bucket on 2026-09-01. Reads are strong (read-your-writes) by default, so the metadata row, the embedded chunk, and the uploaded PDF are visible immediately — no compaction needed.

export BUCKET=blog-research
 
# The bucket exists
dodil data bucket get "$BUCKET"
# ran: status BUCKET_STATUS_ACTIVE, internalSourceId present
 
# The merge-keyed metadata rows upserted (not duplicated)
dodil data sql -b "$BUCKET" \
  "SELECT arxiv_id, title, s3_key FROM papers WHERE primary_category='cs.CL' AND published >= '2026-01-01' ORDER BY published"
# ran → 2 rows: 2601.01234 (Ring Attention), 2603.07890 (Streaming Attention)
 
# The source PDF landed in the object store and a citable URL resolves
dodil data object url papers/2026/2603.07890.pdf -b "$BUCKET" --expires 3600
# ran → https://object.uk-lon-1.dodil.io/blog-research/papers/2026/2603.07890.pdf?X-K3-Token=…  (valid 1h)
 
# Vector search ranks the embedded passages by meaning (cosine distance, lower = closer)
dodil data vsearch -b "$BUCKET" -t paper_chunks --column embedding \
  --text "expanding transformer context window length" --model jina-embeddings-v4 --metric cosine --top-k 4
# ran → c3 0.331, c2 0.427, c1 0.477, c4 0.505 — within the cs.CL-2026 set {c1,c2}, c2 (2603.07890) wins

Live-run notes: the bucket, both tables, four jina-embeddings-v4 embeddings (2048-dim), the metadata upserts, the uploaded PDF, the SQL narrow, the data vsearch rank, and the presigned object url all ran for real and their outputs are captured above. The document-rag recipe path (Step 2 details) is shown as code — data recipe show document-rag confirmed it provisions the collection + ingest rule (no vector store), but its async auto-embed is slower to demo than embedding chunks directly, which is why the searched path is the manual VECTOR(2048) column.

One-shot

Give an agent (with the DODIL MCP connected) this prompt to reproduce the whole tutorial end-to-end:

Stand up an ArXiv research memory bank on DataK3 and answer a question over it, all in one bucket.
 
1. Create a DataK3 bucket named blog-research (description "ArXiv research memory bank").
2. Create a paper_chunks table, merge-keyed on chunk_id, with columns chunk_id, arxiv_id,
   body (string), and embedding VECTOR(2048).
3. Create a papers table, merge-keyed on arxiv_id, with string columns arxiv_id, title,
   primary_category, published, and s3_key.
4. Upload ./2603.07890.pdf to key papers/2026/2603.07890.pdf. Embed its abstract with
   jina-embeddings-v4 and upsert it into paper_chunks (chunk_id c2, arxiv_id 2603.07890).
   Upsert the papers row: 2603.07890, "Streaming Attention for Million-Token Contexts",
   cs.CL, 2026-03-22, papers/2026/2603.07890.pdf.
5. Answer: "find 2026 cs.CL papers about expanding transformer context windows, rank the
   passages by meaning, and give me a citable link to the top paper's PDF." Do the handoff:
   query papers (primary_category='cs.CL' AND published >= '2026-01-01'), run a data vsearch
   on paper_chunks with jina-embeddings-v4, then resolve a 1h pre-signed object url for the
   top PDF.
 
Report the top arxiv_id, its cosine distance, and the citable URL.

Ship it — wrap the loop in a public app

This recipe is a query-and-retrieval loop, not a deployed service. To put it behind a URL your users — or their agents — can call, wrap the loop in an Ignite app and ship it the way every DODIL app ships: DODIL git → CI → a scanned image in the registry → a public endpoint, with versioning and one-command rollback. That full lifecycle is its own tutorial: Ship a DODIL App.

Conclusion

You built a research memory bank where a single question flows narrow → rank → resolve across one DataK3 bucket — the embeddings sit in a VECTOR column right next to the metadata and the raw PDF, no separate vector DB required. Every step runs the same whether you type the CLI or ask your agent.

Next steps:

• Agent Cache — cache the agent's prompt→answer pairs to cut model spend.
• Code & Dev Memory — the same memory pattern for engineering work.
• davidmiheev/research-agent — the live "Atlas" implementation.