What you'll build: a SIEM that keeps petabytes of raw capture cheap in S3, tracks incidents in SQL, catches signature-evading malware by behavioral similarity in the vector pillar, and triages each incident with an Ignite model — writing the verdict back to SQL. One DataK3 bucket, no ETL.

Why it matters. A signature SIEM fights two losing battles this design wins. Cost: indexing petabytes of capture is ruinous, but raw capture sits cold and cheap in S3. Evasion: polymorphic malware changes its signature every run but not its behavior, so a behavioral-similarity search catches the variants a rule DB misses. And instead of a human reading every alert, an Ignite model grades each one — threat, blast radius, priority, next action — so analysts spend their hours on the P1s, not the queue.

What you'll learn:

  • How a VECTOR(2048) column embedded with jina-embeddings-v4 turns behavioral logs into searchable meaning.
  • The alert → evidence → triage → correlate loop across all three pillars plus a model.
  • How to triage one incident by prompting an agent, or a firehose of alerts with an Ignite batch.
  • How to run every step two ways — by prompting an agent over the MCP, or the dodil CLI.

NOTE

Connect the DODIL MCP once — see the two-minute setup. With the dodil MCP server in Claude Code, Cursor, or VS Code you can drive every step by chatting. Each step shows the CLI and an Ask your agent tab.

Problem

A SOC analyst drowns in a signature SIEM's queue: polymorphic malware rotates its signature every run, so rule-DB alerts fire late, in isolation, and never connect the three hosts running the same C2 beacon under three different hashes — while the raw capture needed to confirm it is either priced out of retention or scattered across a separate object store. The payoff of this build is that the analyst opens one alert and immediately sees the fleet-wide behavioral cluster, a link to the exact raw PCAP, and a model-graded verdict (threat type, blast radius, P1/P2/P3), so their hours go to the real P1s instead of triaging noise by hand.

Prerequisites

  • A DODIL organization, and the dodil CLI authenticated (dodil auth login) or the DODIL MCP connected to your agent.
  • A sample capture/log file to ingest.
  • export BUCKET=siem — one bucket is the whole system's data plane.

Scenario

An automated security bot notices an alert in the SQL incident table. It extracts the raw logs from S3, runs a vector search for any other host showing semantically identical (but rule-evading) behavior, has an Ignite model triage the cluster (threat, blast radius, priority, action), and correlates the linked incidents back in SQL — catching obfuscated or polymorphic malware that signature rules miss, and grading it so analysts hit the P1s first.

One bucket, three pillars

DataK3 is one bucket: objects (S3) plus one HTAP tables engine that answers over the same rows by content (SQL, DuckDB dialect) and by meaning (Vector, a VECTOR(dim) column). There is no ETL and no engine to enable — the bucket is ready the moment you create it, and any Postgres/pgvector client can point straight at it (dodil data connect).

PillarHoldsExample keys / columns
S3 (object store)Raw PCAPs (binary evidence) + extracted behavioral logs (text).pcap/…/14-22-09.pcap · logs/…/14-22-09.txt
Vectorjina-embeddings-v4 embeddings of the behavioral log text for similarity-based anomaly detection.incidents.embedding VECTOR(2048)
SQL (table)Incident tickets + the model's triage verdict.incidents(id, host, severity, threat_type, priority, blast_radius, related_to)

Step 1 — Create the bucket

You

Create a DataK3 bucket called siem for raw captures, anomaly vectors, and incidents.

DODIL MCP tools called
data_bucket_create
Agent

Created bucket siem. Its objects (S3) and HTAP tables engine — SQL + vector over the same rows — are ready immediately; no engine to enable.

That's it — the bucket carries both the object store and the tables engine, so SQL tables and vector columns are ready the moment it exists. There is nothing else to provision before Step 2.

Step 2 — Choose the behavioral-indexing path

The vector pillar is not a separate store — it's a VECTOR(2048) column living right on the incidents table, one embedding per behavioral log. We embed the log text with jina-embeddings-v4 (2048-dim); similarity search (dodil data vsearch) then ranks hosts by behavior, catching the variants a signature DB misses. Because the vector sits on the incident row, a look-alike hit is the correlated incident — no join back to a collection.

That is the path this tutorial builds. If instead your logs arrive as a stream of files, the document-rag recipe wires an auto-embed pipeline (embed model + ingest rule) so anything you land under a prefix embeds on its own into a chunks table:

Prefer auto-embed from landed files? Show the recipe path
You

On siem, install the document-rag recipe scoped to the logs/ folder so log text lands and embeds automatically into a searchable table.

DODIL MCP tools called
data_recipe_install
Agent

Installed document-rag on siem: provisioned the jina-embeddings-v4 embed pipeline, a logs_chunks table with a VECTOR column, and an ingest rule scoped to logs/. New log files there embed automatically.

The recipe is the "files land, vectors appear" convenience; the manual VECTOR(2048) column below is the explicit, keyed-to-the-incident path — either way the search is the same dodil data vsearch.

Step 3 — Create the incidents table

One table is the transactional core and the vector index: SQL columns for the ticket, a log_text column for the behavioral summary, an embedding VECTOR(2048) column for its jina-embeddings-v4 vector, and the triage columns the model fills later. It's merge-keyed on id — a PRIMARY KEY is required, so every write is an idempotent upsert.

You

Create an incidents table in siem merge-keyed on id with host, severity, opened_at, status, payload_key, log_text, an embedding VECTOR(2048) column, and triage fields (threat_type, priority, recommended_action, blast_radius, related_to), then open a high-severity incident for host-17.

DODIL MCP tools called
data_table_create→data_table_upsert
Agent

Created the incidents table keyed on id (VECTOR(2048) embedding column + the triage columns the model fills later) and opened incident 4471 (host-17, high, open) pointing at its raw capture key.

TIP

Never write an empty string or null into the key column. A keyed upsert reads ""/null in id back as null and silently drops the row on the next read. Always give id a real value (here, 4471).

Step 4 — Land the capture, its behavioral log, and the embedding

Two objects per capture, one embedded row. The raw PCAP goes to pcap/ — cold, immutable, cheap binary evidence (it's bytes, so it isn't embedded). The extracted behavioral log — a text summary of the flows (Zeek/Suricata notes: connections, DNS queries, beacon timing) — goes to logs/ as cold evidence and is embedded with jina-embeddings-v4 into the incident's embedding column. That split is what makes similarity search work on behavior, not bytes. A date/host key prefix keeps time-window retrieval a cheap prefix object list.

You

Upload the raw capture 14-22-09.pcap to siem under pcap/2026/06/30/host-17/, its behavioral log 14-22-09.txt under logs/2026/06/30/host-17/, then embed the log text with jina-embeddings-v4 and upsert it onto incident 4471's log_text + embedding columns.

DODIL MCP tools called
data_object_create→data_table_upsert
Agent

Uploaded the PCAP (cold evidence) and the behavioral log, embedded the log with jina-embeddings-v4, and merged log_text + the VECTOR(2048) embedding onto incident 4471 — ready for similarity search.

Seed a few more embedded rows the same way — the look-alikes host-24/host-09 and some benign traffic (web browsing, a nightly backup) — so the search has a fleet to rank against.

Step 5 — The threat-hunt loop

SQL surfaces the alert and its capture key, S3 serves the raw evidence, and Vector finds the same behavior elsewhere. dodil data vsearch embeds the query text with the same jina-embeddings-v4 model, then ranks by cosine distance (lower = more similar).

You

In siem, find the newest open high-severity incident, give me a link to its raw capture, and vector-search the incidents' behavioral embeddings for other hosts showing semantically identical beaconing.

DODIL MCP tools called
data_table_query→data_object_url→data_vsearch
Agent

Newest open high-sev: #4471 on host-17. Raw capture (valid 1h): https://object.uk-lon-1.dodil.io/siem/pcap/2026/06/30/host-17/14-22-09.pcap?X-K3-Token=…&X-K3-Expires=… Nearest by behavior: #4473 host-09 (0.28), #4471 host-17 (0.33), #4472 host-24 (0.43) — same jittered C2 beaconing, different signatures — while benign backup/browsing sit far off at 0.54/0.55.

The three beaconing hosts cluster at the top (cosine distance ~0.28–0.43); the benign backup and browsing rows fall away at ~0.54–0.55. That gap is the anomaly signal — no rule matched, yet the behavior grouped itself.

Step 6 — Triage with a model, then correlate back to SQL

The hunt found the neighbors; now close the loop. Have an Ignite model judge the incident — threat type, blast radius, priority, next action — then write the verdict and the linked hosts back to SQL. Two ways, by volume:

  • One incident, mid-hunt → ask your agent: it calls the model over MCP and updates the rows.
  • The alert firehose → an Ignite Python function that HTTP-requests the Models endpoint and triages at scale — the same batch shape as the leads and trading warehouses.
You

Triage incident #4471 in siem from its behavioral log and the look-alike hosts host-24/host-09 using kimi-k2.6 — set threat_type, severity, priority, blast_radius, recommended_action. Update #4471 with the verdict, and open a linked incident for a look-alike host (related_to=4471).

DODIL MCP tools called
ignite_models_chat→data_table_upsert
Agent

Triage: C2 Beaconing (signature-evading), severity high, P1, blast_radius 3, action 'isolate hosts + block rotating C2 domains'. Updated #4471 with the verdict, and opened #4472 (host-24) linked via related_to=4471. The queue now reads as one P1 cluster, not three unlabelled alerts.

For the alert firehose, wrap the same three moves — hunt neighbors → model triage → write back — in an Ignite function and fan it out (with a service account granted ignite.developer + k3.editor, just like the leads classifier):

# triage.py (essence) — an Ignite function; triage a batch of open incidents.
# Same OIDC token -> OpenAI-compatible Models endpoint as the leads/trading batches.
client = OpenAI(base_url="https://api.dodil.io/v1", api_key=dodil_token())
 
def handle(shard):                          # shard payload: {"incident_ids": [4471, 4472, …]}
    for iid in shard["incident_ids"]:
        inc   = sql_one(f"SELECT host, log_text FROM incidents WHERE id={iid}")
        peers = vsearch("incidents", "embedding", inc["log_text"], top_k=20)  # fleet look-alikes
        v     = triage(inc["log_text"], peers)                               # model -> verdict JSON
        upsert_merge("incidents", {"id": iid, **v, "blast_radius": len(peers)})
        for p in peers:
            open_linked_incident(p, related_to=iid)                          # correlate back to SQL
    return {"triaged": len(shard["incident_ids"])}

NOTE

Single vs. firehose — same right-cost logic. For an ad-hoc hunt the agent path is instant; for a stream of alerts, deploy triage.py on Ignite and fan it out (request-invoked, scale-to-zero, with a small always-warm pool if the alert rate is steady). Run the bulk on a cheap model and escalate the severity: critical ones to kimi-k2.6 — spend the good model only where it changes the response.

DataK3 vs. the three-system stack

ConcernThree-system stackDataK3
Petabyte raw retentionObject store separate from analytics; lifecycle driftS3 tier in the same bucket as incidents + vectors
Signature-evading malwareRigid rule DB; misses polymorphic variantsVECTOR(2048) similarity finds "looks like this" across the fleet
Alert → raw evidence → correlationThree systems (Postgres + Pinecone + object store), three auth contextsOne bucket: SQL alert → S3 capture → vector neighbors, one login token

Troubleshooting

  • Embed the log text, not the PCAP bytes. The embedding column holds jina-embeddings-v4 vectors of the behavioral logs; the raw captures under pcap/ stay as cold binary evidence. To embed raw payload vectors directly, compute them yourself and upsert them into the same column.
  • Same model on both sides. data vsearch --text must use the same --model jina-embeddings-v4 that embedded the stored vectors, or the distances are meaningless.
  • Write-backs on a VECTOR table. A plain data table update / SQL UPDATE can fail with point key missing PK column when the table carries a vector. Use data table upsert --merge (partial-column) for the verdict write-back, as in Step 6.
  • Threshold tuning. The cosine-distance cut-off is your false-positive/false-negative dial; tune it against labeled incidents before trusting auto-correlation.

Test

Verify the full loop landed — bucket, embedded logs, the raw evidence, and the triaged incident cluster in SQL. Reads are read-your-writes, so the merges and linked inserts from Step 6 are visible immediately (no compaction step).

export BUCKET=siem
 
# 1. Bucket exists
dodil data bucket get "$BUCKET"
# expect: Name: siem
 
# 2. The behavioral log and raw capture both landed under their prefixes
dodil data object list -b "$BUCKET"
# expect: logs/2026/06/30/host-17/14-22-09.txt  and  pcap/2026/06/30/host-17/14-22-09.pcap
 
# 3. Vector similarity finds the fleet look-alikes for the beaconing behavior
dodil data vsearch -b "$BUCKET" -t incidents --column embedding \
  --text "beaconing to rotating C2 domains with jittered intervals" \
  --model jina-embeddings-v4 --metric cosine --top-k 5
# expect: the beaconing hosts (4471/4472/4473) rank ahead of benign traffic (distance ~0.28–0.43 vs ~0.55)
 
# 4. Incident #4471 carries the model's triage verdict
dodil data sql -b "$BUCKET" \
  "SELECT id, threat_type, priority, blast_radius FROM incidents WHERE id=4471"
# expect: one row — threat_type=c2_beaconing, priority=P1, blast_radius=3
 
# 5. The look-alike host is opened as a linked incident, correlated back to the origin
dodil data sql -b "$BUCKET" \
  "SELECT id, host, related_to FROM incidents WHERE related_to=4471"
# expect: >=1 row — e.g. id=4472 host-24 related_to=4471

One-shot

Hand this to an agent with the dodil MCP connected to reproduce the whole tutorial end-to-end:

Build a one-bucket SIEM on DataK3 and prove it catches signature-evading malware:
 
1. Create a DataK3 bucket `siem` (data_bucket_create).
2. Create an `incidents` table in `siem` merge-keyed on `id` with columns id (long),
   host, severity, opened_at, status, payload_key, log_text, embedding VECTOR(2048),
   threat_type, priority, recommended_action, blast_radius (int), related_to (long) —
   then open incident 4471 for host-17, high, open, payload_key
   pcap/2026/06/30/host-17/14-22-09.pcap (data_table_create, data_table_upsert).
3. Upload the raw capture to pcap/2026/06/30/host-17/14-22-09.pcap and its behavioral
   log to logs/2026/06/30/host-17/14-22-09.txt; embed the log text with
   jina-embeddings-v4 and merge log_text + the embedding onto incident 4471. Seed a
   few more embedded rows (the look-alikes host-24/host-09 plus some benign traffic)
   (data_object_create, ignite_models_embed, data_table_upsert).
4. Run the hunt loop: SQL for the newest open high-severity incident, a pre-signed URL
   for its capture, and data vsearch over incidents.embedding (model jina-embeddings-v4)
   for other hosts showing the same jittered C2 beaconing (data_table_query,
   data_object_url, data_vsearch).
5. Triage #4471 with kimi-k2.6 (threat_type, severity, priority, blast_radius,
   recommended_action), merge the verdict onto #4471, and open a linked incident for a
   look-alike host with related_to=4471 (ignite_models_chat, data_table_upsert).
 
Finish with SQL over incidents where id=4471 and where related_to=4471 to confirm the
triaged P1 cluster reads as one correlated group.

Ship it — wrap the loop in a public app

The hunt loop and its model triage step run interactively here. To put them behind a URL your SOC tools — or an agent — can call, wrap the loop in an Ignite app and ship it the way every DODIL app ships: DODIL git → CI → a scanned image in the registry → a public endpoint, with versioning and one-command rollback. That full lifecycle is its own tutorial: Ship a DODIL App.

Conclusion

One DataK3 bucket takes a security bot from a SQL alert to raw S3 evidence to fleet-wide behavioral neighbors — and a single VECTOR(2048) column, embedded with jina-embeddings-v4, stood up the anomaly search with no separate vector store, CLI or agent.

Next steps: