One Bucket Instead of Postgres + Pinecone + Neo4j: A Customer-360 Pipeline on DODIL

What you'll build: a customer-360 / master-data-management (MDM) pipeline — the kind every data team runs to turn J. Smith @ acme, John Smith, Acme Inc, and Johnny Smith / support into one golden customer — on a single DataK3 bucket. Raw records land once; you resolve duplicates by meaning (embeddings + KNN), fold them into merge-keyed golden records with SQL survivorship, walk the household and corporate family in a graph, and answer the 360 question with SQL + vector + graph over the same rows. Then data connect and watch psql, a pgvector <=> query, and cypher-shell hit that same bucket unchanged.

What you'll learn:

• Model an MDM pipeline as one bucket, five stages — ingest → resolve → golden → graph → query.
• Resolve cross-source duplicates by vector distance — the KNN that proves two spellings are one person.
• Fold survivors into merge-keyed golden records (idempotent — re-run the pipeline safely).
• Build an identity graph over the same rows and traverse the corporate family in one hop.
• Point existing psql / pgvector drivers / cypher-shell at the bucket — drop-in, wire-compatible.

Problem

An enterprise data team owns the customer-360: the single view of who a customer is, stitched from CRM, billing, and support — each of which spells the same person differently and keys them differently. The incumbent way to build it is four systems glued by ETL: a transactional DB (Postgres) for the records, a vector DB (Pinecone) for fuzzy match, a graph DB (Neo4j) for the household / corporate family, and a warehouse for the analytics — with CDC pipelines copying rows between all four. That stack is expensive (four clusters, four bills, four security contexts) and perpetually out of sync: the match you compute in Pinecone lands in Postgres minutes later, the graph lags the warehouse, and a "customer-360" is stale the moment it's assembled.

The payoff of this build: land the rows once and query them three ways — by field (SQL), by meaning (vector), by relationship (graph) — with no copy and no ETL. One bucket, one bill, one auth context, read-your-writes. And because DataK3 speaks the same wires, the psql sessions, pgvector drivers, and Bolt/Neo4j clients your team already runs keep working — you delete the ETL, not the tools.

Prerequisites

• A DODIL organization, and the dodil CLI authenticated (dodil auth login) or the DODIL MCP connected to your agent (Claude Code, Cursor, VS Code, Codex).
• export BUCKET=blog-c360 — one bucket is the whole pipeline's data plane.

Step 1 — Ingest: land raw records into a staging table

The pipeline starts with dirty reality: the same customers, spelled three ways, arriving from crm, billing, and support. Land them once in a merge-keyed staging table. The embedding VECTOR(2048) column is empty for now — Step 2 fills it. A PRIMARY KEY (record_id) makes the write idempotent, so re-ingesting a source is a safe upsert, not a duplicate.

dodil data bucket create "$BUCKET" --description "Customer-360 pipeline — staging, golden, identity graph"
 
dodil data table create stg_customers -b "$BUCKET" \
  --columns-json '[{"name":"record_id","type":"bigint"},{"name":"source","type":"varchar"},{"name":"raw_name","type":"varchar"},{"name":"email","type":"varchar"},{"name":"company","type":"varchar"},{"name":"phone","type":"varchar"},{"name":"embedding","type":"VECTOR(2048)"}]' \
  --merge-key record_id

Now embed each record's identity text (name, company, email) with jina-embeddings-v4 and upsert the row with its vector — so the raw record and its embedding live in one row, one table. (No Pinecone, no customers → embeddings sync job.) Eight source rows, three real people:

# repeated per record — embed the identity text, then upsert the row WITH its vector
VEC=$(dodil ignite models embed jina-embeddings-v4 \
        --input "John Smith, Acme Inc, [email protected]" -o json \
      | jq -c '.data.data[0].embedding')
 
dodil data table upsert stg_customers -b "$BUCKET" \
  --row "$(jq -nc --argjson e "$VEC" '{record_id:1002,source:"billing",raw_name:"John Smith",email:"[email protected]",company:"Acme Inc",phone:"",embedding:$e}')"

After ingest the staging table holds 8 records from 3 sources — the raw material every downstream stage reads:

dodil data sql -b "$BUCKET" \
  "SELECT count(*) AS n, count(DISTINCT source) AS sources FROM stg_customers"
# n  sources
# 8  3

Step 2 — Resolve: find cross-source duplicates by meaning (KNN)

This is the stage a vector DB exists for — except the vectors are already a column in the staging table, so KNN is a query, not a cross-system round-trip. Embed a probe identity and ask for its nearest neighbours. The distances prove the match: the three Acme spellings collapse to a tight cluster while everyone else is far away.

dodil data vsearch -b "$BUCKET" -t stg_customers --column embedding \
  --text "John Smith, Acme Inc, [email protected]" \
  --model jina-embeddings-v4 --metric cosine --top-k 5
# id     score
# 1002   0.0000   ← John Smith / billing
# 1001   0.1383   ← J. Smith / crm
# 1003   0.1729   ← Johnny Smith / support
# 1008   0.5012   ← Bob Chen / billing (different person)
# 1007   0.5171   ← Robert Chen / crm  (different person)

data vsearch returns ids and scores; to capture the match with the fields that justify it, run the same KNN as a pgvector <=> query and join back to the row. This is the artifact you keep as evidence — "J. Smith @ acme" == "John Smith, Acme Inc" == "Johnny Smith / support", with the real distances:

# $LIT = the probe embedding formatted as a pgvector literal '[f1,f2,…]'
dodil data pg -b "$BUCKET" \
  "SELECT record_id, source, raw_name, company, round(embedding <=> '$LIT', 4) AS distance
   FROM stg_customers ORDER BY distance LIMIT 6"
# record_id  source   raw_name      company             distance
# 1002       billing  John Smith    Acme Inc            0.0
# 1001       crm      J. Smith      Acme                0.1383
# 1003       support  Johnny Smith  Acme Incorporated   0.1729
# 1008       billing  Bob Chen      Initech LLC         0.5012   ← well past the 0.25 threshold

Doing the same for the other probes yields three clean clusters — every match under 0.18, every non-match past 0.50 — so a merge threshold of ~0.25 resolves the batch with zero false merges:

Step 3 — Golden records: survivorship into a merge-keyed table

Each resolved cluster becomes one golden row in customers, keyed by customer_id. Survivorship picks the most complete full_name, a canonical email/company, counts the sources it fused, and records the match confidence (1 − worst in-cluster distance). Because customers is merge-keyed, re-running the pipeline upserts the same three rows — the pipeline is idempotent end to end.

dodil data table create customers -b "$BUCKET" \
  --columns-json '[{"name":"customer_id","type":"bigint"},{"name":"full_name","type":"varchar"},{"name":"email","type":"varchar"},{"name":"company","type":"varchar"},{"name":"source_count","type":"int"},{"name":"match_confidence","type":"double"},{"name":"source_records","type":"varchar"}]' \
  --merge-key customer_id
 
dodil data table upsert customers -b "$BUCKET" \
  --row '{"customer_id":1,"full_name":"John Smith","email":"[email protected]","company":"Acme Inc","source_count":3,"match_confidence":0.827,"source_records":"1001,1002,1003"}'
# … same for customer_id 2 (Maria Garcia) and 3 (Robert Chen)

The pipeline just collapsed 8 raw records into 3 golden identities — and every source row is accounted for:

dodil data sql -b "$BUCKET" \
  "SELECT (SELECT count(*) FROM stg_customers)      AS raw_records,
          (SELECT count(*) FROM customers)          AS golden,
          (SELECT sum(source_count) FROM customers) AS collapsed"
# raw_records  golden  collapsed
# 8            3       8

Step 4 — Identity graph: the household and corporate family

Golden records answer "who is this customer" — the graph answers "who are they connected to". Build it over the same bucket: an integer-keyed identity_node table (persons, accounts, and the raw records) and a single identity_edge table carrying a rel column — same_as (record → identity), belongs_to (person → company), parent_of (holding → subsidiary). data pg owns the DDL.

dodil data pg -b "$BUCKET" "CREATE TABLE identity_node (id BIGINT, biz_key VARCHAR, kind VARCHAR, label VARCHAR, PRIMARY KEY (id))"
dodil data pg -b "$BUCKET" "CREATE TABLE identity_edge (src BIGINT, dst BIGINT, rel VARCHAR, PRIMARY KEY (src,dst,rel))"
 
# nodes: golden persons (1,2,3), accounts (100 Acme Holdings, 101 Acme Inc, 102 Globex, 103 Initech), raw records (1001–1008)
dodil data pg -b "$BUCKET" "INSERT INTO identity_node VALUES
 (1,'cust:1','person','John Smith'),(100,'org:acme-holdings','account','Acme Holdings'),
 (101,'org:acme','account','Acme Inc'),(1001,'rec:1001','record','J. Smith / crm'), …"
 
# edges: records --same_as--> identity, person --belongs_to--> account, holding --parent_of--> subsidiary
dodil data pg -b "$BUCKET" "INSERT INTO identity_edge VALUES
 (1001,1,'same_as'),(1002,1,'same_as'),(1003,1,'same_as'),
 (1,101,'belongs_to'),(100,101,'parent_of'), …"
 
# populate FIRST, then snapshot:
dodil data pg -b "$BUCKET" "CREATE GRAPH identity_g NODES (identity_node KEY id) EDGES (identity_edge SRC src DST dst)"

Now walk the corporate family from Acme Holdings (node 100). One undirected traversal descends the whole tree: the subsidiary, the golden identity that belongs to it, and the three raw source records that resolved to that identity — parent_of → belongs_to → same_as in a single query:

dodil data bolt -b "$BUCKET" -g identity_g \
  "MATCH (a)-[*1..3]-(b) WHERE id(a)=100 RETURN b"
# node  hop_distance
# 101   1     ← Acme Inc (the subsidiary)
# 1     2     ← John Smith (golden identity, belongs_to Acme Inc)
# 1003  3     ← Johnny Smith / support   ┐
# 1002  3     ← John Smith / billing     ├ the raw records that resolved to him
# 1001  3     ← J. Smith / crm           ┘
 
# or hydrate direct neighbours with a literal start key, joined to the tables in ONE statement:
dodil data pg -b "$BUCKET" \
  "SELECT n.id, n.kind, n.label FROM graph_neighbors('identity_g', 100) g JOIN identity_node n ON n.id = g.neighbor"
# id   kind     label
# 101  account  Acme Inc

Point the same traversal at a person node (undirected, 1 hop) and you get their household — every raw record that resolved to them, plus their company: MATCH (a)-[*1..1]-(b) WHERE id(a)=1 RETURN b → 1003, 1002, 1001, 101.

Step 5 — The Ignite resolver + the Models adjudication gate

Steps 1–4 are the pipeline; running it continuously is an Ignite job. The resolver is a real handler.py — embed the incoming record, KNN against stg_customers, and if the best distance clears the threshold, write the golden customers row and the same_as graph edge. An Ignite app is a separate workload, so it gets its own service account (client-credentials → bearer), least-privilege roles, and the creds injected as runtime env:

dodil auth service-account create c360-resolver-sa
dodil auth service-account grant-role $SA_UUID ignite-authorization-service ignite.developer
dodil auth service-account grant-role $SA_UUID k3-authorization-service k3.editor
 
dodil ignite app deploy c360-resolver --code ./resolver --runtime python \
  --env DODIL_SERVICE_ACCOUNT_ID=$SA_ID --env DODIL_SERVICE_ACCOUNT_SECRET=$SA_SECRET
 
dodil ignite invoke c360-resolver \
  --payload '{"record_id":1009,"source":"billing","raw_name":"J Smith","email":"[email protected]","company":"Acme Inc."}'

The handler (resolver/handler.py) — mints its own token, embeds, KNN-matches, and writes both the golden row and the graph edge back to the bucket over the DataK3 HTTP API:

 
 
ID   = "https://id.dodil.io/realms/dodil/protocol/openid-connect/token"
DATA = "https://api.dodil.io/data/v1"          # DataK3 HTTP API
MODELS = "https://api.dodil.io/v1"             # OpenAI-compatible
BUCKET, THRESHOLD = "blog-c360", 0.25
 
def token():
    r = requests.post(ID, data={
        "grant_type": "client_credentials",
        "client_id": os.environ["DODIL_SERVICE_ACCOUNT_ID"],
        "client_secret": os.environ["DODIL_SERVICE_ACCOUNT_SECRET"]})
    r.raise_for_status()
    return r.json()["access_token"]
 
def handler(payload, ctx):
    tok = token()
    H = {"Authorization": f"Bearer {tok}"}
    text = f'{payload["raw_name"]}, {payload["company"]}, {payload["email"]}'
 
    # 1) embed the incoming identity
    emb = requests.post(f"{MODELS}/embeddings", headers=H,
        json={"model": "jina-embeddings-v4", "input": text}).json()
    vec = emb["data"][0]["embedding"]
 
    # 2) KNN against staging (pgvector literal) — nearest existing identity
    lit = "[" + ",".join(f"{x:.6f}" for x in vec) + "]"
    hit = requests.post(f"{DATA}/{BUCKET}/sql", headers=H, json={"sql":
        f"SELECT record_id, embedding <=> '{lit}' AS d FROM stg_customers "
        f"ORDER BY d LIMIT 1"}).json()["rows"][0]
    if float(hit["d"]) > THRESHOLD:
        return {"matched_customer_id": None, "distance": hit["d"]}
 
    cid = resolve_customer_id(hit["record_id"])   # cluster → golden customer_id
 
    # 3) write the golden row (merge = idempotent upsert; rows are JSON strings)
    requests.post(f"{DATA}/{BUCKET}/tables/customers/merge", headers=H,
        json={"rows": [json.dumps({"customer_id": cid, "source_count": 1})]})
    # 4) write the same_as identity edge (re-CREATE the graph to pick up new edges)
    requests.post(f"{DATA}/{BUCKET}/tables/identity_edge/merge", headers=H,
        json={"rows": [json.dumps(
            {"src": payload["record_id"], "dst": cid, "rel": "same_as"})]})
    return {"matched_customer_id": cid, "distance": hit["d"], "edge_written": True}

The Models gate. Clean clusters auto-merge; the ambiguous pairs — distance in the gray zone between auto-merge and auto-reject — are where MDM projects burn analyst-hours. Hand those to a model. kimi-k2.6 adjudicates merge vs. keep-separate and you write the verdict to a match_decisions audit table:

dodil ignite models chat kimi-k2.6 --message \
  'Entity-resolution adjudicator. Distance 0.31 (auto-merge 0.20, auto-reject 0.45). Same person?
   A: "John Smith, Acme Inc, [email protected]"  B: "Jon Smith, Acme LLC, [email protected]".
   Reply ONLY compact JSON: {"decision":"merge"|"keep_separate","confidence":0-1,"reason":"<12 words"}'
# {"decision":"merge","confidence":0.8,"reason":"Nickname, similar employer, and email patterns match."}
 
dodil data table upsert match_decisions -b "$BUCKET" \
  --row '{"pair_id":"1002:x-acmellc","record_a":"John Smith, Acme Inc","record_b":"Jon Smith, Acme LLC","distance":0.31,"decision":"merge","confidence":0.8,"reason":"Nickname, similar employer, and email patterns match."}'

Query it — one bucket, three pillars, drop-in clients

The customer-360 question — "who is Acme's golden contact, what other spellings did we fuse, and who sits above them in the corporate tree?" — is one SQL read + one graph walk over the same rows. No join across systems, no staleness:

dodil data sql -b "$BUCKET" \
  "SELECT customer_id, full_name, email, company, source_count, match_confidence
   FROM customers ORDER BY match_confidence DESC"
# customer_id  full_name     email                    company   source_count  match_confidence
# 3            Robert Chen   [email protected]        Initech   2             0.9
# 2            Maria Garcia  [email protected]         Globex    3             0.851
# 1            John Smith    [email protected]  Acme Inc  3             0.827

Now the drop-in proof. data connect prints endpoints for the same bucket — DB name = bucket, credential = your login token — and the psql / pgvector / Bolt clients your team already runs point straight at it, no driver changes:

dodil data connect "$BUCKET" -o psql   # → postgresql://token:[email protected]:5432/blog-c360
dodil data connect "$BUCKET" -o env    # → DATA_PG_URL, DATA_BOLT_URI, DATA_BOLT_DB
 
# 1) plain psql — the SQL pillar, unchanged tooling
psql "$DATA_PG_URL" -c "SELECT full_name, company FROM customers ORDER BY match_confidence DESC"
 
# 2) pgvector <=> over the SAME rows — the vector pillar, standard driver
psql "$DATA_PG_URL" -c \
  "SELECT record_id, raw_name FROM stg_customers ORDER BY embedding <=> '$LIT' LIMIT 3"
 
# Both wires are TLS. Until the next CLI release, `data connect` still prints the pre-TLS forms
# (`?sslmode=prefer`, `neo4j+s://`) — override them, or you connect in the clear (or not at all):
DATA_PG_URL="${DATA_PG_URL/sslmode=prefer/sslmode=require}"
DATA_BOLT_URI="bolt+s://bolt.uk-lon-1.dodil.io:7687"   # neo4j+s asks for routing; this plane refuses it
 
# 3) cypher-shell / any Bolt client — the graph pillar
cypher-shell -a "$DATA_BOLT_URI" -d blog-c360 \
  "MATCH (a)-[*1..3]-(b) WHERE id(a)=100 RETURN b"

DataK3 vs. the multi-system stack

This is the whole thesis in one table. The customer-360 you just built is a transactional DB + a vector DB + a graph DB + a warehouse + the CDC between them — collapsed into one bucket where each pillar is a query surface over one copy of the rows, and the client that used to talk to each system still does.

One bucket, one bill, one auth context. The match you compute is visible to the next SQL read immediately (read-your-writes), so the 360 is never stale — there is no window between systems for it to be stale in.

Test

Runnable end-to-end assertions — this pipeline was validated live on blog-c360 on 2026-09-01 (tested: true). Everything below ran against live DataK3; the Ignite deploy in Step 5 is shown as runnable code (the resolver handler.py is complete and correct) — the resolve → golden → graph → gate logic it automates was each executed live by hand.

# 1) Ingest landed 8 records from 3 sources
dodil data sql -b "$BUCKET" "SELECT count(*) AS n, count(DISTINCT source) AS s FROM stg_customers"
# expect: n = 8, s = 3
 
# 2) Resolve — the Acme trio clusters below 0.18, non-matches past 0.50 (Step 2 table)
dodil data vsearch -b "$BUCKET" -t stg_customers --column embedding \
  --text "John Smith, Acme Inc, [email protected]" --model jina-embeddings-v4 --metric cosine --top-k 3
# expect: 1002 ~0.00, 1001 ~0.138, 1003 ~0.173
 
# 3) Golden — 8 raw records collapsed to 3 identities, all accounted for
dodil data sql -b "$BUCKET" \
  "SELECT (SELECT count(*) FROM stg_customers) AS raw, (SELECT count(*) FROM customers) AS golden,
          (SELECT sum(source_count) FROM customers) AS collapsed"
# expect: raw = 8, golden = 3, collapsed = 8
 
# 4) Graph — the corporate family under Acme Holdings
dodil data bolt -b "$BUCKET" -g identity_g "MATCH (a)-[*1..3]-(b) WHERE id(a)=100 RETURN b"
# expect nodes: 101 (hop 1), 1 (hop 2), 1001/1002/1003 (hop 3)
 
# 5) Gate — the model returns a JSON verdict
dodil ignite models chat kimi-k2.6 --message '…adjudicate the ambiguous pair, reply compact JSON…'
# expect: {"decision":"merge"|"keep_separate","confidence":…,"reason":"…"}

One-shot

On a fresh DataK3 bucket blog-c360, build a customer-360 pipeline:
1. Create a merge-keyed staging table stg_customers(record_id bigint, source, raw_name, email,
   company, phone, embedding VECTOR(2048)). Ingest 8 customer records from sources crm/billing/support
   with varied spellings for 3 real people (Acme, Globex, Initech). For each, embed "name, company,
   email" with jina-embeddings-v4 and upsert the row with its vector inline.
2. Resolve duplicates: for each identity, KNN over the embedding column (pgvector <=> / data vsearch,
   cosine). Confirm each cluster is < 0.18 and non-matches > 0.50; a 0.25 threshold separates them.
3. Golden: create merge-keyed customers(customer_id, full_name, email, company, source_count,
   match_confidence, source_records) and upsert one survivorship row per cluster. Assert 8 raw → 3 golden.
4. Identity graph: build identity_node(id BIGINT KEY, biz_key, kind, label) and identity_edge(src, dst,
   rel) with same_as/belongs_to/parent_of edges (persons, accounts incl. an Acme Holdings parent, raw
   records). Populate fully, then CREATE GRAPH identity_g. Traverse the family from Acme Holdings.
5. Ignite resolver (handler.py): own service account (ignite.developer + k3.editor), embed → KNN → write
   golden row + same_as edge. Models gate: kimi-k2.6 adjudicates ambiguous pairs into match_decisions.
6. data connect -o psql|env and query the same bucket with psql, a pgvector <=> SQL, and cypher-shell.

Ship it — c360-resolver as a public 360 API (git → CI → registry → deploy)

The c360-resolver above resolves identities as code; as a public endpoint it becomes the Customer-360 API your apps query for a golden record — shipped through the real DODIL supply chain, no external CI or PaaS. (The whole lifecycle, including versioning and rollback, is validated end to end in Ship a DODIL App; this is that pipeline for c360-resolver.)

1 — Source in DODIL git. Create the repo, mint a git.editor key, push the app (handler.py + Dockerfile + requirements.txt):

dodil git repo create c360-resolver
DK=$(dodil auth apikey issue --service git --role git.editor --name c360-resolver-push -o json | jq -r '.secret // .key')
git init && git add . && git commit -m "ship c360-resolver"
git push "https://x:$DK@git.dodil.io/$ORG/c360-resolver.git" main

2 — CI → a scanned image in the DODIL registry.

dodil ignite build create c360-resolver --git-url "https://git.dodil.io/$ORG/c360-resolver.git" \
  --tag "registry.dodil.io/$ORG/c360-resolver:v1"
dodil registry vuln c360-resolver v1

3 — Deploy public and curl it. Build-on-deploy needs no registry pull secret:

dodil ignite app deploy c360-resolver --git-url "https://git.dodil.io/$ORG/c360-resolver.git" \
  --dockerfile-path Dockerfile --allow-unauthenticated --port 8080 --health-path /healthz
BASE=$(dodil ignite app get c360-resolver --output json | jq -r '.public_urls[0]')
curl -s "https://$BASE/healthz"    # 200, no token

Roll out a v2 and roll back with dodil ignite version rollback c360-resolver 1 — the full versioning walk-through is in Ship a DODIL App.

Conclusion

You built a full customer-360 / MDM pipeline — ingest, entity resolution by meaning, survivorship golden records, an identity graph, and a model-gated adjudicator — on one DataK3 bucket with no ETL between systems, and proved that existing psql / pgvector / Bolt clients hit it unchanged. The transferable skill: resolve entities by vector distance and fold them into merge-keyed golden records with a graph of the survivors — three pillars, one copy of the rows.

From here, wire a domain on top of the same pattern: the CRM on DODIL turns these golden contacts into a cost-gated GTM pipeline, and the AI-powered SIEM applies the same vector-similarity core to catch behavior a signature DB misses.