What you'll build: a real leads data warehouse — one DataK3 bucket that's your store from the
first org. Stand up the warehouse, discover companies from a free source into it, enrich
each with its homepage, classify ~200,000 of them on a low-cost model into a queryable
organizations table, then open only the qualified ones into a people table. Then two moves that
only a one-bucket engine makes cheap: propagate qualification across corporate families with a
graph, and dedup near-duplicate orgs with vector similarity — all queried with SQL over the
same rows. This is the pipeline we actually run for our own lead targeting, and an agent wrote most of
it by prompting the DODIL MCP.
Why classify first? Because opening a lead — revealing its contacts on a data provider — costs credits. At 200k companies, revealing everyone (most of whom will never buy) burns a fortune. So you classify cheaply first, on the lowest-cost model that clears the bar, and only pay to enrich the leads that actually fit. The classifier is a cost gate, and that's the whole game.
NOTE
The product you're selling here is a stand-in — a generic data-infrastructure product for AI-native teams. Swap in your own product and ICP; the pipeline doesn't change.
What you'll learn:
- The backbone pattern: stand up a DataK3 warehouse → discover → enrich → classify on a low-cost model → open qualified contacts → query. Everything lives in one bucket — objects for the raw trail, tables for the queryable rows.
- Three pillars over one copy of the rows: SQL for the funnel, a graph to carry a parent's qualification down to its subsidiaries, and vectors to catch orgs that are the same company under two domains — no second system, no ETL.
- Why low-cost, accurate classification is what keeps a lead funnel affordable — always route bulk work to the lowest-cost model that does the job.
- How to run every DODIL step two ways — by prompting an agent over the MCP, or the
dodilCLI.
The problem — from 200k orgs to a qualified pipeline
A demand-gen team buys a list, enriches it, scores it, and emails it — and every one of those verbs lives in a different tool with a different copy of the same company. The enrichment vendor knows the headcount, the scoring tool knows the fit, the sequencer knows who replied, and the warehouse knows what closed. Nobody can answer "which of these 200,000 organizations is worth paying to qualify?" without a join across four vendors and a week of someone's time.
The money is in that question. Enriching and scoring a list costs real per-record spend, so qualifying everything is how the budget disappears before the good leads are found.
Everything lands in one DataK3 bucket from org #1 — raw objects for the audit trail, tables
(organizations, people, plus a node/edge pair for the graph and a VECTOR column for dedup) for the
queryable warehouse.
| Stage | Lands in | Cost | Tool |
|---|---|---|---|
| 1. Discover orgs | DataK3 objects raw/orgs/ | free (0 credits) | Hunter.io Discover |
| 2. Enrich homepages | DataK3 objects raw/homepages/ | free | parallel scrape → snippet |
| 3. Classify (the gate) | organizations table | low cost | Ignite Models (kimi-k2.6) |
| 4. Open qualified | people table | paid | Hunter Domain Search / agent |
| 5. Link families | org_node / org_edge graph | free | graph traversal → propagate fit |
| 6. Dedup lookalikes | org_embeddings VECTOR(2048) | cents | jina-embeddings-v4 KNN |
| 7. Query & score | — | — | DataK3 SQL / graph / vector over the bucket |
Discovery and enrichment stay free, classification stays cheap — because the expensive step (revealing contacts / opening leads on the data provider) comes last, and only for the leads classification already qualified. Get the classifier wrong and you pay to open 200k companies; get it right and you pay for ~19k.
NOTE
Connect the DODIL MCP once — see the two-minute setup. With the dodil MCP
server in Claude Code, Cursor, or VS Code, an agent can write and run this whole pipeline from
prompts — which is how we built it. Each DODIL step shows an Ask your agent tab and the CLI.
Prerequisites
- A DODIL organization, and the
dodilCLI authenticated (dodil auth login) or the DODIL MCP connected to your agent. - A free Hunter.io API key (
HUNTER_API_KEY) for discovery. export BUCKET=dodil-leads— one bucket is the whole warehouse's data plane.
Step 1 — Stand up the warehouse
Create the store first. It's a single DataK3 bucket with two core tables: organizations (one row
per company — the queryable warehouse) and people (contacts, filled later, only for qualified
buyers). Raw discovery and enrichment land as objects in this same bucket; the tables hold the
classified, queryable rows. A PRIMARY KEY (merge-key) is required — writes are keyed, so re-runs
and shard retries upsert idempotently. There is no engine to enable and no --freshness to set: reads
are read-your-writes.
Create a DataK3 bucket dodil-leads, then create an organizations table keyed on domain (domain, organization, country, headcount_band, layer, category, ai_native, data_intensity, buyer, angle, description as string; fit_score as double; node_id as long; emails_count as int; verified as boolean) and a people table keyed on email (email, domain, full_name, first_name, last_name, position, department, seniority, verification, linkedin, source as string; hunter_confidence as double).
data_bucket_create→data_table_createCreated dodil-leads with organizations (15 cols, key domain) and people (12 cols, key email). The bucket is your store from org #1 — raw objects go under raw/, these tables hold the queryable rows.
export BUCKET=dodil-leads
dodil data bucket create "$BUCKET" --description "Leads warehouse: raw objects + orgs/people + graph + vectors"
# organizations — one row per company (short names — see dodil://k3/enums)
dodil data table create organizations -b "$BUCKET" \
--columns-json '[
{"name":"domain","type":"string","nullable":false},
{"name":"organization","type":"string","nullable":true},
{"name":"country","type":"string","nullable":true},
{"name":"headcount_band","type":"string","nullable":true},
{"name":"layer","type":"string","nullable":true},
{"name":"category","type":"string","nullable":true},
{"name":"ai_native","type":"string","nullable":true},
{"name":"data_intensity","type":"string","nullable":true},
{"name":"buyer","type":"string","nullable":true},
{"name":"fit_score","type":"double","nullable":true},
{"name":"angle","type":"string","nullable":true},
{"name":"description","type":"string","nullable":true},
{"name":"node_id","type":"long","nullable":true},
{"name":"emails_count","type":"int","nullable":true},
{"name":"verified","type":"boolean","nullable":true}
]' \
--merge-key domain
# people — contacts (filled later, only for qualified buyers)
dodil data table create people -b "$BUCKET" \
--columns-json '[
{"name":"email","type":"string","nullable":false},
{"name":"domain","type":"string","nullable":true},
{"name":"full_name","type":"string","nullable":true},
{"name":"first_name","type":"string","nullable":true},
{"name":"last_name","type":"string","nullable":true},
{"name":"position","type":"string","nullable":true},
{"name":"department","type":"string","nullable":true},
{"name":"seniority","type":"string","nullable":true},
{"name":"hunter_confidence","type":"double","nullable":true},
{"name":"verification","type":"string","nullable":true},
{"name":"linkedin","type":"string","nullable":true},
{"name":"source","type":"string","nullable":true}
]' \
--merge-key emailTIP
Never write an empty string or null into a key column. A keyed upsert reads "" and null
back as null and silently drops the row on the next read. Use a non-empty sentinel ("none",
"n/a") — that's why the people rows below set linkedin to "none", not "".
Step 2 — Discover organizations → DataK3 objects (free, 0 credits)
The trick that keeps this free: drive Hunter's Discover endpoint with structured filters
(industry × country × headcount band), not the natural-language query — company-level discovery
costs 0 credits and never reveals (or bills for) emails. One request per segment tags every
returned company with a reliable {industry, country, headcount_band}; dedupe by domain, and land the
result straight into the bucket as objects — no local file.
# discover.py (essence) — many small segment requests, 0 credits, resumable
payload = {
"industry": {"include": [industry]}, # exact hunter industry name
"headquarters_location": {"include": [{"country": country}]}, # ISO alpha-2, e.g. "GB"
"headcount": [band], # "1-10", "11-50", "51-200", ...
"company_type": {"exclude": ["educational", "government agency", "non profit"]},
}
# POST https://api.hunter.io/v2/discover?api_key=... → dedupe by domain, then write each org to DataK3:
# raw/orgs/<segment>.jsonl {domain, country, headcount_band}Run it across your ICP matrix — for this example that's EMEA AI builders (GB/IE/DE/FR/NL/… × AI/data-infra/dev-tools × 1–200 employees). A few hundred segment requests yields ~200,000 unique orgs, each cleanly tagged and landed in the bucket:
Upload my discovered orgs from ./raw/orgs to the dodil-leads bucket under raw/orgs/.
data_object_createUploaded ~200k discovered orgs under raw/orgs/ in dodil-leads — the raw trail, each tagged with domain/country/headcount_band, ready to enrich. Nothing left a local file.
# land the discovered orgs in the bucket (the raw trail — not a local JSON)
dodil data object create ./raw/orgs -b "$BUCKET" --key raw/orgs --recursiveTIP
Verify the server echoed back every filter you sent before trusting the tags — a silently-dropped
filter would mis-label the batch. (Our discover.py skips any response whose filters don't match.)
Step 3 — Enrich homepages → DataK3 objects (free, no LLM)
Give the classifier something to read. Scrape each domain's homepage in parallel and keep a compact
snippet — title, meta/OG description, <h1>s, and a slice of body text. Merge each snippet with its
discovery tags and write it back to the bucket, so Step 4 has everything it needs to write a complete
org row. No model, no cost.
# fetch_homepages.py (essence) — 30-way parallel, resumable, ~600 chars/company
head = " | ".join([title, meta_or_og_description, first_two_h1s])
snippet = (head + " || " + stripped_body)[:600]
# merge the discovery tags + snippet, write to DataK3:
# raw/homepages/<segment>.jsonl {domain, country, headcount_band, snippet}Upload my enriched homepages from ./raw/homepages to the dodil-leads bucket under raw/homepages/.
data_object_createUploaded the enriched homepages under raw/homepages/ — each record carries the discovery tags plus a 600-char snippet, so the classifier can write a complete org row in one pass.
dodil data object create ./raw/homepages -b "$BUCKET" --key raw/homepages --recursiveStep 4 — Classify on low-cost inference → the organizations table
This is the money step. Two hundred thousand companies is far too many to hand-label — and running a
frontier model on every one would be ruinous. So route the batch to Ignite Models, pointed at
kimi-k2.6: a strong, low-cost model reached through the same OpenAI-compatible endpoint.
Low-cost tokens, quality that matches a frontier model — the lowest-cost option that clears the bar.
IMPORTANT
Why classification quality is the whole game. Discovery is free — but the next step, revealing a contact's email or opening a lead on the data platform, costs money per lead. At 200,000 orgs, opening everything would cost a fortune, and most of that spend would land on companies that will never buy. So classification is the gate: qualify buyers first, and you only ever pay to open leads that actually fit. A sloppy classifier doesn't just lose deals — it burns real credits revealing tens of thousands of unqualified companies. That's exactly why it's worth making the low-cost inference accurate: it protects the expensive step downstream.
Give it the org + snippet and ask for strict JSON — layer, category, ai_native,
data_intensity, buyer, fit_score, angle:
Classify this company as a lead for our product with kimi-k2.6 — org: Greyparrot, greyparrot.ai, 'AI waste analytics; computer vision scans waste streams on sorting facilities in real time, dashboards + APIs for recyclers.' Return JSON: layer, category, ai_native, data_intensity, buyer, fit_score, angle.
ignite_models_chat{"layer":"app_data_ai","category":"Waste Analytics","ai_native":"core","data_intensity":"high","buyer":"strong","fit_score":0.9,"angle":"Real-time waste CV requires scalable data infra"}
dodil ignite models chat kimi-k2.6 \
--system 'Classify a company as a sales lead for our product — a data-infrastructure platform for AI-native software teams. Return ONLY compact JSON with keys layer (app_data_ai|app_light|devtools|infra|services), category, ai_native (core|feature|none), data_intensity (high|med|low), buyer (strong|possible|weak), fit_score (0-1), angle (<=12 words).' \
--message 'org: Greyparrot | domain: greyparrot.ai | snippet: AI waste analytics — computer vision scans waste streams on sorting facilities in real time; dashboards and APIs for recyclers.'That's a real response from kimi-k2.6 on DODIL — and it matches what a human (and a frontier
model) would say: app_data_ai, AI-native core, a strong buyer at 0.9 fit. Now do it 200,000
times.
NOTE
Right-cost model per job. Reserve frontier models for reasoning; bulk classification belongs on
a low-cost, fast model. Swapping is a string on the same endpoint — one key, one bill. Always
route bulk work to the lowest-cost model that does the job. For pure label extraction you can go even
cheaper with a task model like gliner-multi-v2.1 via ignite models infer.
Run the batch on Ignite → upsert organizations
Two hundred thousand classifications run fine from a laptop overnight — but to finish in minutes (and re-run on a schedule), fan the loop out on Ignite serverless: shard the enriched-homepage objects, deploy a classifier that handles one shard, and invoke one execution per shard. It scales 0 → hundreds and back to zero, so you pay for the burst, not a standing box.
The classifier runs headless — it calls Ignite Models and writes DataK3 with no human at a browser — so first give it a service account scoped to exactly that: invoke models (Ignite) and write DataK3.
# secret is shown ONCE — capture it now
dodil auth service-account create ci-leads-classify -o json
dodil auth service-account grant-role $SA_UUID ignite-authorization-service ignite.developer
dodil auth service-account grant-role $SA_UUID k3-authorization-service k3.editor
# expose the id/secret to the function as runtime env vars (in your app config):
# DODIL_SERVICE_ACCOUNT_ID / DODIL_SERVICE_ACCOUNT_SECRETEach execution reads its shard of homepage objects from DataK3, classifies each org on the Models
OpenAI-compatible endpoint, and upserts the full row into organizations. Hundreds of executions
hammering the endpoint will get rate-limited (HTTP 429), so the per-call path retries with backoff.
# classify.py (essence) — an Ignite handler; one shard of homepage-object keys per execution.
from openai import OpenAI, RateLimitError
# Headless auth: exchange the service-account creds for an OIDC access token, then
# point a standard OpenAI client at the DODIL Models API (OpenAI-compatible).
def dodil_token():
r = requests.post("https://id.dodil.io/realms/dodil/protocol/openid-connect/token", data={
"grant_type": "client_credentials",
"client_id": os.environ["DODIL_SERVICE_ACCOUNT_ID"],
"client_secret": os.environ["DODIL_SERVICE_ACCOUNT_SECRET"]})
return r.json()["access_token"]
client = OpenAI(base_url="https://api.dodil.io/v1", api_key=dodil_token())
MODEL = "kimi-k2.6" # low-cost, frontier-quality labels for the bulk pass
def classify(snippet, tries=6):
for i in range(tries):
try:
r = client.chat.completions.create(
model=MODEL, response_format={"type": "json_object"},
messages=[{"role": "system", "content": SCHEMA},
{"role": "user", "content": snippet}])
return json.loads(r.choices[0].message.content)
except RateLimitError: # 429 -> exponential backoff + jitter
time.sleep(min(2**i, 30) + random.random())
raise RuntimeError("model retries exhausted")
def handler(payload, ctx): # payload: {"keys": ["raw/homepages/seg-03.jsonl", ...]}
rows = []
for key in payload["keys"]:
for o in data_read_jsonl(key): # {domain, country, headcount_band, snippet}
rows.append({"domain": o["domain"], "country": o["country"],
"headcount_band": o["headcount_band"], **classify(o["snippet"])})
data_table_upsert("organizations", rows) # upsert, merge on domain -> re-run is safe
return {"classified": len(rows)}Then shard the homepage object keys, deploy, and fan out one execution per shard:
Deploy my classify function to Ignite under the ci-leads-classify service account, then fan it out across all homepage shards to classify the whole 200k batch and upsert organizations.
ignite_app_deploy→ignite_invokeDeployed classify (headless via ci-leads-classify → kimi-k2.6 over the OpenAI-compatible endpoint → upserts organizations). Fanned 198,417 orgs across ~330 parallel executions; ~6 minutes, then scaled to zero. organizations now holds the classified rows.
# 1) shard the enriched-homepage object keys (a few files per shard)
# object list has no server-side prefix filter — list all keys, filter to raw/homepages/ in shard.py
dodil data object list -b "$BUCKET" -o json | ./shard.py # -> shards/*.json
# 2) deploy the classifier — it reads DODIL_SERVICE_ACCOUNT_* from its runtime env
dodil ignite app deploy classify --code ./classify --runtime python \
--env DODIL_SERVICE_ACCOUNT_ID=$SA_ID --env DODIL_SERVICE_ACCOUNT_SECRET=$SA_SECRET
# 3) fan out — one execution per shard, in parallel (0 -> hundreds, scale-to-zero)
for shard in shards/*.json; do
dodil ignite invoke classify --payload-file "$shard" &
done; waitIMPORTANT
Tune the fan-out to the rate limit. Backoff absorbs bursts, but if every execution retries at once you've built a thundering herd. Cap concurrency within each shard and size the shard count so total in-flight calls sit just under your model quota — saturate the limit, don't trip it. The domain-merged upsert makes a re-run safe.
NOTE
Running it on a schedule? Keep a warm pool. The one-off backfill is bursty, so scale-to-zero is ideal — you pay only for the burst. But when you re-run this on a regular schedule, a function that has scaled to zero cold-starts on every tick: the first calls eat container startup latency. For a recurring pipeline, deploy the classifier with a small always-warm pool so each scheduled run lands on hot pods:
dodil ignite app deploy classify --code ./classify --runtime python \
--auto-min-instances 1 --auto-predictive--auto-min-instances holds N pods warm between runs; --auto-predictive pre-warms ahead of
expected load. Size the warm pool to your schedule's concurrency — enough to skip cold starts,
without keeping so much idle capacity that you lose the scale-to-zero savings between ticks.
Step 5 — Open qualified leads → the people table
Now, and only now, the paid step: reveal contacts for the leads classification already qualified.
Revealing a company's contacts is a Hunter Domain Search (it consumes credits, unlike free
Discover), so pull the SQL list of strong buyers, reveal contacts for just those, and upsert them
into people (merge on email). The easy way is to ask your agent:
For every strong buyer in dodil-leads, reveal decision-maker contacts via Hunter Domain Search and insert them into the people table.
data_table_query→data_table_upsertRevealed contacts for 19,004 strong buyers — and skipped the ~180k that didn't qualify, which is the spend you just avoided. Upserted 41,206 people rows (merged on email). Greyparrot → Anne-Sophie Ribeiro, Director of Revenue Operations, verified 98%.
# 1) the qualified list — strong buyers only (the gate)
dodil data table query "SELECT domain FROM organizations WHERE buyer = 'strong'" -b "$BUCKET"
# 2) reveal contacts for just those domains (Hunter Domain Search — the paid call),
# then upsert into people (merge-key email, so re-runs don't duplicate).
# NOTE: linkedin is "none", never "" — an empty key/value can read back as null.
dodil data table upsert people -b "$BUCKET" \
--row '{"email":"[email protected]","domain":"greyparrot.ai","full_name":"Anne-Sophie Ribeiro","first_name":"Anne-Sophie","last_name":"Ribeiro","position":"Director of Revenue Operations","department":"operations","seniority":"executive","hunter_confidence":98.0,"verification":"hunter_verified","linkedin":"none","source":"hunter_domain_search"}'Prefer to automate it? The same gate as a short script — the SQL picks the qualified, Hunter reveals, DataK3 upserts:
# reveal.py (essence) — open ONLY the qualified, upsert into people
strong = sql("SELECT domain FROM organizations WHERE buyer='strong'") # the gate: ~19k, not 200k
for domain in strong:
contacts = hunter_domain_search(domain, os.environ["HUNTER_API_KEY"]) # PAID — credits spent here
data_table_upsert("people", [normalize(c, domain) for c in contacts]) # upsert, merge on emailNOTE
This is where classification pays for itself. You revealed ~19k qualified companies instead of 200k — the ~180k you skipped are the credits the classifier saved. Filter tightly (department Engineering/Executive, seniority Senior/Director/Executive) to spend only on the people who actually pick infrastructure.
Step 6 — Link corporate families → propagate qualification with a graph
A flat organizations table misses a structural truth about B2B targeting: companies come in
families. A holding group owns subsidiaries; a strong parent almost always means the whole portfolio
standardizes infrastructure together. If you only score each domain in isolation, a subsidiary that
alone looks like a lukewarm possible gets dropped — even though its parent already qualified and
would happily roll it into the same deal.
DataK3 gives you a graph pillar over the same rows to fix exactly this. Model corporate structure as
a node table + a single edge table (with a rel column for parent_of / same_group), then traverse
the family to carry a parent's qualification down to its subsidiaries — and to spot orgs that are
really the same group under two domains.
IMPORTANT
A graph node KEY must be an integer. String business keys (a domain) are rejected, so project each
org to an integer node_id in a dedicated org_node table. And CREATE GRAPH snapshots its edges
at creation time — populate org_node and org_edge fully first, then create the graph. (v1
graphs are one node table + one edge table; put the relationship kind in a rel column.)
Build the node and edge tables with data pg (it owns the graph DDL), populate them from the enriched
orgs, then create the graph:
In dodil-leads, build a corporate-family graph: an org_node table (integer id KEY + domain + name) and an org_edge table (src, dst, rel), populate them from the classified orgs and known ownership, then CREATE GRAPH orgfamily.
data_pgorg_node (6 rows) and org_edge (4 edges: parent_of ×2, same_group ×2) created; GRAPH orgfamily created over org_node KEY id / org_edge SRC src DST dst.
# integer-keyed node table (project each org's domain -> node_id)
dodil data pg -b "$BUCKET" "CREATE TABLE org_node (id BIGINT PRIMARY KEY, domain VARCHAR, name VARCHAR)"
dodil data pg -b "$BUCKET" "INSERT INTO org_node VALUES
(1,'greyparrot.ai','Greyparrot'),(2,'primetag.com','PrimeTag'),(3,'predictx.com','PredictX'),
(10,'greyparrot-group.com','Greyparrot Group'),(11,'recyclense.io','Recyclense'),
(12,'grey-parrot.co.uk','Grey Parrot UK')"
# single edge table — the relationship kind lives in rel (parent_of | same_group)
dodil data pg -b "$BUCKET" "CREATE TABLE org_edge (src BIGINT, dst BIGINT, rel VARCHAR, PRIMARY KEY (src,dst))"
dodil data pg -b "$BUCKET" "INSERT INTO org_edge VALUES
(10,1,'parent_of'),(10,11,'parent_of'),(1,12,'same_group'),(12,1,'same_group')"
# snapshot the graph AFTER the edges exist
dodil data pg -b "$BUCKET" "CREATE GRAPH orgfamily NODES (org_node KEY id) EDGES (org_edge SRC src DST dst)"Now the payoff traversal: from the parent group (node 10, a strong buyer), walk the family and
join straight back to organizations — in one statement — to see which subsidiaries the parent's
qualification should pull in. Recyclense scored only possible (0.55) on its own, but it's a
subsidiary of a strong parent, so it belongs on the list:
From the Greyparrot Group node in dodil-leads, who is in the corporate family within 2 hops, and what did each score on its own? Join graph_neighbors to organizations.
data_pg→data_boltFamily of Greyparrot Group (node 10): greyparrot.ai (Greyparrot, strong, 0.9) and recyclense.io (Recyclense, possible, 0.55). Recyclense alone looked lukewarm — but the parent is strong, so ICP-fit propagates and it joins the outreach list.
# graph_neighbors('graph', start_key) — a literal start key joins to the SQL tables in ONE statement
dodil data pg -b "$BUCKET" \
"SELECT g.neighbor, o.organization, o.domain, o.buyer, o.fit_score
FROM graph_neighbors('orgfamily', 10) g
JOIN org_node n ON n.id = g.neighbor
JOIN organizations o ON o.domain = n.domain
ORDER BY g.neighbor"
# -> 1 Greyparrot greyparrot.ai strong 0.9
# 11 Recyclense recyclense.io possible 0.55 <- propagate the parent's qualification
# same traversal in Cypher over Bolt (node keys come back as BIGINT; join org_node for names)
dodil data bolt -b "$BUCKET" -g orgfamily "MATCH (a)-[*1..2]->(b) WHERE id(a)=10 RETURN b"
# -> node 1 (hop 1), node 11 (hop 1), node 12 (hop 2, via same_group)NOTE
This is a targeting decision the flat table can't make. Propagating a strong parent's fit down to
possible subsidiaries surfaces qualified accounts you'd otherwise skip — and the same_group edges
(Greyparrot ↔ Grey Parrot UK) flag two domains that are one company, so you don't open (and pay for)
both. The graph turns isolated rows into an account map.
TIP
v1 Cypher can't filter on a relationship property (the rel column) — only on id. Traverse the
whole family in Cypher/graph_neighbors, then filter rel in the enclosing SQL (JOIN org_edge … WHERE rel='parent_of') when you need just one kind of link.
Step 7 — Dedup lookalikes with vectors
Discovery pulls the same company under more than one domain (greyparrot.ai and grey-parrot.co.uk),
under slightly different names, sometimes in different countries. Open both and you pay twice; count
both and your funnel is inflated. Exact string matching won't catch them — but meaning will.
DataK3's vector pillar is just a VECTOR(2048) column on a table in the same bucket — no separate
store. Embed each org's description with jina-embeddings-v4, then KNN-search finds near-duplicates
and lookalike segments by cosine distance:
Create an org_embeddings table in dodil-leads with a VECTOR(2048) column, embed each org's description with jina-embeddings-v4, and upsert one row per org.
data_table_create→ignite_models_embed→data_table_upsertCreated org_embeddings (domain key + organization + VECTOR(2048)); embedded 6 org descriptions with jina-embeddings-v4 and upserted one row each (partial --merge, one vector per call).
dodil data table create org_embeddings -b "$BUCKET" \
--columns-json '[
{"name":"domain","type":"string","nullable":false},
{"name":"organization","type":"string","nullable":true},
{"name":"embedding","type":"VECTOR(2048)","nullable":true}
]' \
--merge-key domain
# embed each description, then upsert ONE vector row per call (2048-dim rows are large):
# vec=$(dodil ignite models embed jina-embeddings-v4 --input "<org description>" -o json | jq -c '.data.data[0].embedding')
# dodil data table upsert org_embeddings -b "$BUCKET" --merge \
# --row "{\"domain\":\"greyparrot.ai\",\"organization\":\"Greyparrot\",\"embedding\":$vec}"TIP
Write vector rows one at a time, with upsert --merge. Batching many 2048-dim vectors in one
call can hit a gRPC frame-size limit, and a plain UPDATE/data table update on a table that has a
VECTOR column can fail (point key missing PK column) — the partial --merge upsert is the reliable
write-back path.
Now find the near-duplicates. data vsearch embeds your query text with the same model and returns the
closest orgs by cosine distance — the near-dup surfaces right behind the query itself:
In dodil-leads, find orgs whose description is nearest to Greyparrot's — the near-duplicates to merge.
data_vsearchgreyparrot.ai 0.0000, grey-parrot.co.uk 0.0608 (the near-duplicate — same company, second domain), recyclense.io 0.3449, greyparrot-group.com 0.3491. The 0.06 hit is a merge/dedup candidate; the ~0.34 hits are same-sector lookalikes, not dups.
dodil data vsearch -b "$BUCKET" -t org_embeddings --column embedding \
--text "AI waste analytics; computer vision scans waste streams on sorting facilities in real time, dashboards and APIs for recyclers." \
--model jina-embeddings-v4 --metric cosine --top-k 4
# id score
# greyparrot.ai 0.0000 <- the query org itself
# grey-parrot.co.uk 0.0608 <- near-duplicate: same company, merge these
# recyclense.io 0.3449 <- same sector, distinct company (a lookalike, not a dup)
# greyparrot-group.com 0.3491A tight distance (≈0.06) is a dedup candidate — collapse the two domains onto one org before you spend credits opening both. The mid-range hits (≈0.34) are lookalike segments: same market, different company — exactly the seed list for "find me more orgs like our best customers."
Step 8 — Query the warehouse
Here's the payoff, and it's just SQL over the one bucket — no exports, no second system, no compaction step before a JOIN (reads are read-your-writes). Count the funnel, rank the buyers, and pull the outreach list.
In dodil-leads: how many orgs are strong vs total, the sharpest AI-native targets by fit_score, and the ready-to-contact list of executive contacts at strong buyers joined from organizations and people.
data_sqlFunnel: 198,417 orgs → 19,004 strong (9.6%). Sharpest: greyparrot.ai (0.9), primetag.com (0.9), greyparrot-group.com (0.88), grey-parrot.co.uk (0.87), predictx.com (0.83) — all app_data_ai / core. Ready-to-contact: 12,880 execs at strong buyers with a verified email — e.g. Anne-Sophie Ribeiro (Greyparrot, Director of Revenue Operations). That's your outreach list, straight from SQL.
# the funnel in one line — qualified vs total
dodil data sql -b "$BUCKET" \
"SELECT buyer, COUNT(*) AS n FROM organizations GROUP BY buyer ORDER BY n DESC"
# the sharpest targets — AI-native, data-heavy, highest fit
dodil data sql -b "$BUCKET" \
"SELECT domain, organization, fit_score, angle FROM organizations
WHERE layer = 'app_data_ai' AND ai_native = 'core'
ORDER BY fit_score DESC LIMIT 25"
# the outreach list: senior contacts at strong buyers (join orgs + people — no compaction needed)
dodil data sql -b "$BUCKET" \
"SELECT o.organization, o.country, p.full_name, p.position, p.email, o.angle
FROM organizations o JOIN people p ON p.domain = o.domain
WHERE o.buyer = 'strong' AND p.seniority IN ('executive','director')
ORDER BY o.fit_score DESC LIMIT 50"Query it — one bucket, three pillars, drop-in clients
The same rows answer by content (SQL — the funnel), by relationship (graph — the corporate
family), and by meaning (vector — the dedup) with no ETL and no second copy. And you don't have to
go through the dodil CLI: any Postgres/pgvector or Bolt/Neo4j client points straight at the
bucket. data connect prints the endpoints — DB name = the bucket, credential = your login token:
Print the drop-in connection endpoints for the dodil-leads bucket so I can point psql and cypher-shell at it.
data_connectpg: postgresql://token:…@pg.uk-lon-1.dodil.io:5432/dodil-leads (paste into psql / any pgvector client). bolt: bolt+s://bolt.uk-lon-1.dodil.io:7687, DB dodil-leads (cypher-shell / any Neo4j driver).
dodil data connect "$BUCKET" -o psql # postgresql://token:…@pg.uk-lon-1.dodil.io:5432/dodil-leads
dodil data connect "$BUCKET" -o env # DATA_PG_URL + DATA_BOLT_URI/DATA_BOLT_DB for scripts
# Both wires are TLS: use ?sslmode=require and bolt+s://. Until the next CLI release `data connect`
# still prints the pre-TLS forms (?sslmode=prefer, neo4j+s://) — rewrite them before you connect.
# then, with no DODIL client at all:
psql "$(dodil data connect "$BUCKET" -o psql | sed 's/sslmode=prefer/sslmode=require/')" -c \
"SELECT organization, fit_score FROM organizations WHERE buyer='strong' ORDER BY fit_score DESC LIMIT 5"DataK3 vs. the multi-system stack
The same warehouse on a conventional stack is four systems and the glue between them. One DataK3 bucket is one store, one bill, one auth context:
| Job in this warehouse | Conventional stack | On DataK3 |
|---|---|---|
| Orgs/people funnel (SQL) | Postgres / a warehouse | organizations + people tables (DuckDB dialect) |
| Corporate-family links | Neo4j (separate DB + sync) | org_node/org_edge + CREATE GRAPH — same bucket |
| Dedup / lookalike search | Pinecone / pgvector service | VECTOR(2048) column + data vsearch — same rows |
| Raw discovery/homepage trail | S3 + a catalog | DataK3 objects raw/… in the same bucket |
| Classification at scale | a model API + your own queue | Ignite Models (kimi-k2.6) + serverless fan-out |
| Drop-in client access | per-system drivers & creds | one data connect — psql / bolt / pgvector |
| Keeping them in sync | ETL / CDC pipelines | none — one copy of the rows, read-your-writes |
Test
Everything below ran live against DataK3 (org IHDIASH, region uk-lon-1) on a blog-leads validation
bucket — the funnel counts and 198k/19k figures in the prose are the production numbers, but the
schema, the kimi-k2.6 classify, the graph traversal, and the vector KNN are the real outputs shown
here. The Ignite fan-out deploy is shown as code (the batch runs on your own service account).
# core is queryable
dodil data sql -b "$BUCKET" "SELECT count(*) AS n FROM organizations" # -> n = 6 (validation set)
# the graph traversal propagates a parent's qualification to a subsidiary
dodil data pg -b "$BUCKET" \
"SELECT g.neighbor, o.buyer, o.fit_score FROM graph_neighbors('orgfamily',10) g
JOIN org_node n ON n.id=g.neighbor JOIN organizations o ON o.domain=n.domain"
# -> 1 strong 0.9 ; 11 possible 0.55 (Recyclense pulled in by strong parent Greyparrot Group)
# the vector KNN flags the near-duplicate domain
dodil data vsearch -b "$BUCKET" -t org_embeddings --column embedding \
--text "AI waste analytics computer vision on sorting facilities, dashboards for recyclers" \
--model jina-embeddings-v4 --metric cosine --top-k 2
# -> greyparrot.ai 0.0000 ; grey-parrot.co.uk 0.0608 (merge candidate)
# one real low-cost classify
dodil ignite models chat kimi-k2.6 --message 'org: Greyparrot | greyparrot.ai | AI waste analytics'
# -> {"layer":"app_data_ai","ai_native":"core","buyer":"strong","fit_score":0.9, ...}One-shot: build it by prompting your agent
Prefer to skip the step-by-step? With the DODIL MCP connected, paste this one prompt — a precise spec an agent can build and run end-to-end:
Build a leads data warehouse on DODIL DataK3 + Ignite and run the classify batch. Confirm each step.
1. Create a DataK3 bucket `dodil-leads` and tables in it:
• `organizations`, merge-keyed on `domain` — domain, organization, country, headcount_band, layer,
category, ai_native, data_intensity, buyer, angle, description (string); fit_score (double);
node_id (long); emails_count (int); verified (boolean).
• `people`, merge-keyed on `email` — email, domain, full_name, first_name, last_name, position,
department, seniority, verification, linkedin, source (string); hunter_confidence (double).
• never write "" or null into a key column — use a sentinel like "none".
2. My discovered orgs and enriched homepages are already objects under raw/orgs/ and raw/homepages/
(each homepage record: {domain, country, headcount_band, snippet}).
3. Write a Python Ignite handler `classify` and deploy it with a service account (roles ignite.developer
+ k3.editor). It reads a shard {"keys":[...]}, calls DODIL Models (OpenAI-compatible, base_url
https://api.dodil.io/v1, bearer = client-credentials token) with model kimi-k2.6, asks for ONLY JSON
(layer|category|ai_native|data_intensity|buyer|fit_score|angle), retries 429 with backoff, and
upserts {domain,country,headcount_band,...result} into organizations.
4. List raw/homepages/ keys, shard them, fan out one Ignite execution per shard.
5. Build a corporate-family graph: org_node(id BIGINT KEY, domain, name) + org_edge(src,dst,rel), fully
populate both, then CREATE GRAPH orgfamily. Traverse graph_neighbors('orgfamily', <parent_id>) joined
to organizations to propagate a strong parent's qualification to its subsidiaries.
6. Add org_embeddings(domain KEY, organization, embedding VECTOR(2048)); embed each org description with
jina-embeddings-v4 (one row per upsert --merge) and KNN with data vsearch to flag near-duplicates.
7. Show the funnel (strong vs total), the sharpest app_data_ai/core targets, and the family/dedup results.The single knob is the classify model (kimi-k2.6) at step 3. Because every table is merge-keyed, the
batch upserts rather than duplicates — safe to re-run or widen.
Ship it — make classify a public endpoint
The classify app above is shown as code. Ship it the last mile so users hit a real URL, not a snippet —
the managed runtime compiles your handler.py and returns a public FQDN:
Deploy the classify app publicly with no auth and give me its URL.
ignite_app_deploy→ignite_app_getDeployed classify (deployment_state: deployed). Its public FQDN on ignite.dodil.cloud is callable with no token — or call it with dodil ignite invoke classify.
dodil ignite app deploy classify --code ./classify --runtime python --allow-unauthenticated
dodil ignite app get classify --output json # → public_urlsThat's the quick path (managed compile). The full supply chain — DODIL git → CI checks → a scanned image in the DODIL registry → versioning and one-command rollback — is its own tutorial: Ship a DODIL App.
Conclusion
One DataK3 bucket that's the store from org #1 — free discovery and enrichment as objects, a low-cost
model (kimi-k2.6) for the heavy classification into a queryable organizations table, a paid
reveal gated on the classifier for people, a graph that propagates a parent's qualification
across its corporate family, and a vector column that dedups lookalike orgs — all queried by
content, relationship, and meaning over one copy of the rows. A real leads pipeline that's low-cost,
fast, and entirely on DODIL — and it re-skins straight into customer-service triage, invoice intake, or
contract review.
Next steps:
- Inside the DODIL AI Cloud — the backbone agents offload to.
- Trading-Signal Warehouse — the same backbone, pointed at markets.