What you'll build: an FRB-hunting store that prunes petabytes to an exact calibration cohort with partitioned SQL, ranks candidates by signal-shape similarity over the rendered spectrograms, and serves the raw array — all in one DataK3 bucket.

What you'll learn:

  • How partitioned SQL tables prune huge datasets before any similarity work.
  • How to embed rendered spectrogram images for shape similarity into a VECTOR(2048) column with jina-embeddings-v4 — and when to let the image-rag recipe wire the embed pipeline for you.
  • How to run every step two ways — by prompting an agent over the MCP, or the dodil CLI.

NOTE

Connect the DODIL MCP once — see the two-minute setup. With the dodil MCP server in Claude Code, Cursor, or VS Code you can drive every step by chatting. Each step shows the CLI and an Ask your agent tab.

Problem

A radio astronomer hunting fast radio bursts is drowning: a single dish writes petabytes of raw signal arrays, but only a handful were captured in the exact calibration state that makes a detection trustworthy — right target, right frequency, right dish — and among those, only the ones whose spectrogram shape matches a dispersed transient are worth a human's time. Doing this the usual way means stitching an object store, a warehouse, and a vector DB together by hand and hoping they stay in sync. The payoff of doing it in one DataK3 bucket: partitioned SQL prunes the firehose to the calibration cohort before any similarity work runs, vector search ranks that cohort by signal shape, and a pre-signed link serves the raw array for analysis — so the researcher goes from petabytes to a ranked shortlist of real candidates in three commands, reproducibly.

Prerequisites

  • A DODIL organization, and the dodil CLI authenticated (dodil auth login) or the DODIL MCP connected to your agent.
  • A raw observation file (e.g. obs-88123.npy) and its rendered spectrogram image (obs-88123.png).

Scenario

An astrophysicist hunting fast radio bursts (FRBs):

  1. Writes a SQL query to select data gathered while the dish pointed at Andromeda, calibrated to 1.4 GHz.
  2. Uses Vector search to find spectrograms whose shape resembles a known FRB.
  3. Pulls the raw signal array from object storage to analyze the anomaly.

Isolate the signal from the noise by combining exact calibration state with fuzzy signal-shape similarity over huge raw payloads.

How the three pillars map

One DataK3 bucket is objects plus an HTAP tables engine — SQL and Vector answer over the same rows, no ETL and no engine to enable. Here is how the FRB store lands across them:

PillarHoldsExample keys / columns
ObjectsRaw signal arrays (cold) + rendered spectrogram images (provenance).raw/…/obs-88123.npy · spectrograms/…/obs-88123.png
VectorEmbeddings of the rendered spectrograms (jina-embeddings-v4) for shape similarity.signals.embedding VECTOR(2048)
SQL (table)Strictly relational equipment + calibration state at capture time.observations(obs_id, target, freq_ghz, dish, calibrated_at, s3_key)

Step 1 — Create the bucket

You

Create a DataK3 bucket called blog-physics for raw telemetry, signal vectors, and calibration state.

DODIL MCP tools called
data_bucket_create
Agent

Created bucket blog-physics. The tables engine is ready and an internal object source is wired for ingestion.

The bucket is ready the moment it is created — one bucket is objects and the tables engine, so there is nothing to enable before you write tables.

Step 2 — Model the calibration state

Calibration is strictly relational — and partitioned by dish so the SQL prune is cheap. A PRIMARY KEY (merge-key) is required: writes are keyed, so re-runs and shard retries upsert idempotently.

You

Create an observations table in blog-physics keyed on obs_id, partitioned by dish, with target, freq_ghz, calibrated_at, and s3_key; then upsert the M31/dish-A cohort plus a couple of off-cohort rows.

DODIL MCP tools called
data_table_create→data_table_upsert
Agent

Created observations partitioned by dish, keyed on obs_id, and upserted 6 rows (3 in the M31/dish-A/1.4 GHz cohort, 3 off-cohort).

TIP

Never write an empty string or null into a key column. Both read back as null and silently drop the row on the next read. Give every obs_id a real, non-empty value.

Step 3 — Index the spectrograms for shape similarity

Similar signals look similar: an FRB's dispersed sweep has a characteristic shape in the time–frequency spectrogram. So render each observation to a spectrogram image, embed it with jina-embeddings-v4 (2048-dim, multimodal), and land the vector in a VECTOR(2048) column on a signals table. There is no separate "vector store" — a vector is just a column over the same rows.

You

In blog-physics create a signals table keyed on obs_id with morphology, png_key, and a VECTOR(2048) embedding column; embed each rendered spectrogram with jina-embeddings-v4 and upsert one row per observation.

DODIL MCP tools called
data_table_create→ignite_models_embed→data_table_upsert
Agent

Created signals with a VECTOR(2048) column; embedded 3 spectrograms with jina-embeddings-v4 and upserted them (one row per call).

The embedding column is what similarity search ranks over. Because it lives on the same table as obs_id, a hit joins straight back to the calibration row and its s3_key — no cross-system lookup.

Alternative: let image-rag wire the embed pipeline for you

Rather than embedding by hand, the image-rag recipe provisions an embed pipeline + an ingest rule so that spectrograms dropped under a folder are embedded on upload — one command instead of the embed-then-upsert loop:

dodil data recipe install image-rag -b "$BUCKET"
# then upload rendered spectrograms under the scoped folder and they embed automatically

The manual VECTOR(2048) path above is the right choice when you want to embed with your own featurization (dedispersion, DM–time, matched filtering) and control exactly which vector lands per observation; the recipe is the right choice when a stream of images should just embed themselves.

Step 4 — Land the raw array + its rendered spectrogram

Two objects per observation. The raw signal array goes to raw/ — cold, immutable, the analysis truth (a .npy; binary, not embedded). The rendered spectrogram image goes to spectrograms/ as provenance for the vector you embedded in Step 3.

You

Upload the raw array obs-88123.npy to blog-physics under raw/2026-06-30/dish-A/, and its rendered spectrogram obs-88123.png under spectrograms/2026-06-30/dish-A/.

DODIL MCP tools called
data_object_create
Agent

Uploaded the raw array to raw/2026-06-30/dish-A/obs-88123.npy (cold) and the rendered spectrogram to spectrograms/2026-06-30/dish-A/obs-88123.png.

TIP

To render spectrograms or feed raw arrays to a compute job, mount the bucket: dodil data mount "$BUCKET" /mnt/astro — the arrays appear as local files for NumPy/analysis code.

Step 5 — The FRB hunt (prune → rank → resolve)

Partitioned SQL cuts petabytes to the exact cohort before any similarity work; Vector ranks that cohort by shape; a pre-signed link serves the raw truth. Reads are read-your-writes, so the rows and objects you just wrote are visible immediately — no compaction step.

You

In blog-physics, select M31 observations from dish-A at 1.4 GHz, rank the spectrograms by similarity to a high-DM broadband transient, and give me a link to the top candidate's raw array.

DODIL MCP tools called
data_table_query→data_vsearch→data_object_url
Agent

Cohort: 3 M31/dish-A/1.4 GHz observations. Closest FRB-shaped spectrogram: obs-88123 (cosine distance 0.070), then obs-88155 (0.223), obs-88140 (0.369). Raw array (valid 1h): https://object.uk-lon-1.dodil.io/blog-physics/raw/2026-06-30/dish-A/obs-88123.npy?X-K3-Token=…

The prune returns exactly the three M31/dish-A/1.4 GHz rows — obs-88200 (0.8 GHz), obs-88300 (dish-C) and obs-77001 (Crab) never reach the ranker. Then Vector orders that cohort by shape: obs-88123 (the dispersed sweep) is closest, and the flat RFI comb obs-88140 is furthest — the exact triage a human wants.

DataK3 vs. the three-system stack

ConcernThree-system stackDataK3
Petabyte raw + exact calibration + similarityObject store + warehouse + vector DB, kept in sync by handOne bucket; partitioned SQL prunes before Vector ranks
"Same calibration state AND similar signal"Cross-system join, easy to get subtly wrongSQL filter and the VECTOR(2048) column live on the same rows
Feeding raw arrays to computeBespoke staging from object storeMount the bucket (FUSE/NFS) into the analysis job
Drop-in clientsThree sets of drivers + credsdodil data connect "$BUCKET" prints one Postgres/pgvector endpoint (DB = bucket)

Troubleshooting

  • Rendered image vs. rigorous featurization. Embedding the spectrogram image description (jina-embeddings-v4) catches shape similarity and runs today; for dedispersion / DM–time features, compute your own vectors offline and upsert them into the same VECTOR(2048) column — the caller fully controls what lands per observation.
  • Upsert VECTOR rows one per call. Batching many 2048-dim vectors in a single upsert can hit a gRPC "first frame too large" limit. Embed, then upsert row-by-row (as in Step 3).
  • Maintenance at scale. Huge partitioned tables benefit from periodic dodil data table optimize … --z-order-column …; dodil data table compact is optional performance maintenance only (reads are already read-your-writes). Partition around your dominant query (target/frequency/dish).
  • Mount needs privileges. FUSE/NFS mount starts a daemon and may require elevated permissions.

Test

Verify the outcome end to end against the real blog-physics bucket. Reads are read-your-writes by default, so the upserted rows and uploaded objects are visible immediately — no compaction needed.

export BUCKET=blog-physics
 
# 1. The bucket exists
dodil data bucket get "$BUCKET"
# expect: bucket "blog-physics" returned (not a not-found error)
 
# 2. observations is partitioned by dish and keyed on obs_id
dodil data table describe observations -b "$BUCKET"
# expect: columns obs_id/target/freq_ghz/dish/calibrated_at/s3_key; partition column = dish; merge key = obs_id
 
# 3. The calibration cohort query resolves the three M31/dish-A/1.4 GHz rows
dodil data table query \
  "SELECT obs_id, s3_key FROM observations
   WHERE target='M31' AND freq_ghz=1.4 AND dish='dish-A' ORDER BY obs_id" \
  -b "$BUCKET"
# expect: obs-88123, obs-88140, obs-88155 (off-cohort rows excluded)
 
# 4. The signals table carries a VECTOR(2048) embedding column
dodil data table describe signals -b "$BUCKET"
# expect: embedding column typed VECTOR(2048); merge key = obs_id
 
# 5. Both objects landed — raw array and rendered spectrogram
dodil data object show raw/2026-06-30/dish-A/obs-88123.npy -b "$BUCKET"
dodil data object show spectrograms/2026-06-30/dish-A/obs-88123.png -b "$BUCKET"
# expect: object metadata (size > 0), not a not-found error
 
# 6. Shape search ranks the FRB-shaped spectrogram first
dodil data vsearch -b "$BUCKET" --table signals --column embedding \
  --text "dispersed single-pulse broadband sweep, high dispersion measure" \
  --model jina-embeddings-v4 --metric cosine --top-k 5
# expect: obs-88123 closest (cosine distance ≈ 0.07), obs-88140 (RFI) furthest
 
# 7. A pre-signed URL for the raw array is issued
dodil data object url raw/2026-06-30/dish-A/obs-88123.npy -b "$BUCKET" --expires 3600
# expect: an https URL on object.uk-lon-1.dodil.io valid for ~1h

One-shot

Hand this to an agent connected to the DODIL MCP to reproduce the whole tutorial end to end:

Using the DODIL MCP, build an FRB-hunting store in one DataK3 bucket:
 
1. Create a DataK3 bucket called `blog-physics` for raw telemetry, signal vectors, and
   calibration state (data_bucket_create).
2. Create an `observations` table in `blog-physics` keyed on obs_id, partitioned by dish, with
   columns obs_id (varchar), target (varchar), freq_ghz (double), dish (varchar),
   calibrated_at (varchar), s3_key (varchar) — merge-key obs_id, partition-column dish
   (data_table_create). Upsert the M31/dish-A/1.4 GHz cohort (obs-88123, obs-88140, obs-88155)
   plus a few off-cohort rows (data_table_upsert).
3. Create a `signals` table keyed on obs_id with morphology, png_key, and an embedding
   VECTOR(2048) column (data_table_create). Embed each rendered spectrogram's morphology with
   jina-embeddings-v4 (ignite_models_embed) and upsert one row per observation (data_table_upsert).
4. Upload the raw array to raw/2026-06-30/dish-A/obs-88123.npy (cold) and its rendered
   spectrogram to spectrograms/2026-06-30/dish-A/obs-88123.png (data_object_create).
5. Run the FRB hunt: query M31 / dish-A / 1.4 GHz observations (data_table_query), rank the
   signals embeddings against "dispersed single-pulse broadband sweep, high dispersion measure"
   with jina-embeddings-v4, cosine, top-k 5 (data_vsearch), and return a pre-signed URL for the
   top candidate's raw array valid for one hour (data_object_url).
 
Then verify with the ## Test section: the bucket exists, observations is partitioned by dish and
keyed on obs_id, the cohort query returns the three M31/dish-A rows, signals carries a VECTOR(2048)
column, both objects landed, and the shape search ranks obs-88123 closest.

Ship it — wrap the hunt in a public app

The prune → rank → resolve hunt runs interactively here. To put it behind a URL your pipeline — or an agent — can call, wrap the hunt in an Ignite app and ship it the way every DODIL app ships: DODIL git → CI → a scanned image in the registry → a public endpoint, with versioning and one-command rollback. That full lifecycle is its own tutorial: Ship a DODIL App.

Conclusion

Partitioned SQL prunes the firehose to the exact calibration cohort, an embedded VECTOR(2048) column ranks candidates by signal shape, and a pre-signed link serves the raw array — reproducible science in one DataK3 bucket, CLI or agent.

Next steps:

  • AI SIEM — the same firehose-plus-similarity shape for security telemetry.
  • Multimodal E-Commerce — raw payload + metadata + similarity, commercial edition.