What you'll build: a problem-management engine that turns recurring incidents into a single, root-caused problem record. It clusters related incidents by vector similarity over the incidents.embedding column, asks a kimi-k2.6 gate to synthesise the root cause and a permanent fix, publishes a known-error (KEDB) row, and links a change for the fix — one Ignite app over the same DataK3 bucket your ITSM already lives in (see the ITSM core). It consumes core incidents/problems/changes and adds two tables of its own.

The problem — and why it matters

The on-call raises max_connections on the order database at 2am. Two weeks later a different engineer kills the leaked cursors. A month after that, a third restarts the ETL worker. Each incident gets closed, each fix is real — and nobody ever asks "why does this keep happening?" because the three tickets are three rows in a queue that no human reads side by side. That is the difference between incident management (restore service now) and problem management (make it stop recurring). Skipping the second one is why the same outage bills MTTR three times.

The blocker has always been finding the recurrence. Symptoms are free text — "connections maxed out", "pool exhausted", "max_connections reached under load" are the same problem in three wordings, and no WHERE short_description LIKE … catches all three. You need to cluster by meaning, not by string.

What collapses onto one bucket: the incidents are already embedded (incidents.embedding, jina-embeddings-v4, the "have we seen this before?" index the core built), so clustering is a cosine KNN over the same rows — no export to a vector DB. The root-cause synthesis is a kimi-k2.6 gate on the same auth context. The problem, its incident links, the known-error, and the permanent-fix change are all SQL rows in the same bucket — one JOIN away from each other. The payoff: a recurring outage becomes one auditable problem record with a published workaround and a change already in flight, instead of an infinite loop of identical incidents.

PieceLands inPillar / runs on
Cluster membershiptable problem_incidentsSQL
The known-error recordtable known_errorsSQL (the KEDB)
The recurrence signalKNN over core incidents.embeddingVector (jina-embeddings-v4, 2048-dim)
The root-cause synthesiskimi-k2.6 verdict → known_errors / problemsIgnite Models (the gate)
The clustering enginereads incidents, writes problems/problem_incidents/known_errors/changesIgnite app itsm-problem-clusterer

NOTE

Connect the DODIL MCP once, then every step shows an Ask your agent tab (the default — DODIL is agent-native) and a CLI tab. Install itsm/core first if you're building the full suite — it owns the masters (incidents with its embedding, problems, changes) this skill reads. Standalone, Step 0 stubs and seeds those masters, so you can run this end to end without the rest of the ITSM.

Prerequisites

  • The dodil CLI (dodil auth login) or the DODIL MCP connected to your agent.
  • export BUCKET=itsm — the same bucket your ITSM masters live in. (itsm is four characters, so unlike the GL's two-character gl it needs no -suite suffix; DataK3 requires a bucket name of at least three.)
  • The live model ids (confirm with dodil ignite models list): chat kimi-k2.6, embeddings jina-embeddings-v4 (2048-dim). Both confirmed live 2026-09-08.
  • The itsm/core masters incidents (with the embedding VECTOR(2048) column), problems, changes. If you don't have them, Step 0 stubs the minimum this skill reads.

Step 0 — Stub + seed the masters you consume (skip if you have itsm/core)

This skill reads the ITSM masters; it doesn't own them. If you built itsm/core (or the full suite), those tables already exist and are populated — skip to Step 1. Standalone, create incidents (with its embedding column), problems, and changes with the same column definitions itsm/core uses (every non-key column nullable:true, so a partial row never trips NotNullViolation), then seed a demo cluster: the three resolved pg-orders-primary connection-exhaustion incidents that keep recurring, the fresh one, plus an unrelated incident so clustering has something to exclude.

You

Create the itsm bucket, then three merge-keyed masters with all non-key columns nullable: incidents (key id: number, short_description, description, ci_id(long), service_id(string), state, priority, category, assignment_group, problem_id(long), opened_at, resolved_at, resolution, embedding VECTOR(2048)); problems (key id: number, short_description, description, root_cause, state, known_error(boolean), workaround, related_change_id(long), created_at); changes (key id: number, ci_id(long), short_description, description, type, state, risk, requested_by, assignment_group, approval_state, problem_id(long)).

DODIL MCP tools called
data_bucket_create→data_table_create
Agent

Created bucket itsm and 3 masters — incidents (pk id, embedding VECTOR(2048), service_id VARCHAR), problems (pk id), changes (pk id) — all non-key columns nullable. These are the itsm/core masters; if you already ran itsm/core they're here and this step is a no-op.

The one-line model change that broke every write in this component

Look again at Change.type. Until the seven ITSM components were stood up on one bucket, this package mapped that column to a Python attribute called change_type:

change_type: Mapped[str | None] = mapped_column("type", String, nullable=True)   # DON'T

It looks harmless. type shadows a builtin, change_type reads better, SQLAlchemy explicitly supports the rename. And it is wrong twice over, because two idioms this entire codebase is built on take column names, not attribute names:

  • db.upsert() is handed a dict of columns and builds INSERT … ON CONFLICT from them. Pass it {"change_type": "normal"} and you get an "unconsumed column names" error — there is no change_type column. Pass it {"type": "normal"} and the ORM never sees it either.
  • The read-merge-upsert idiom that every back-link in the suite uses — {c.name: getattr(row, c.name) for c in Model.__table__.columns} — walks the columns and asks the instance for each one by name. The moment one attribute is named differently, it raises AttributeError: 'Change' object has no attribute 'type'.

So the canonical rule, now stated in the package's own models.py: the ORM attribute name IS the column name. A nicer Python name is not worth it. The column name is the contract — with SQL, with db.upsert, and with every other component reading the same table. (And the usual excuse doesn't apply here: type is not a DuckDB reserved word. at is, which is why timestamps in this suite are opened_at/event_at and never at. The rename bought nothing and cost an afternoon.)

This is one of six integration bugs the single-bucket run turned up, and it is worth being blunt about how it survived: this component's own ## Test passed with change_type in place, every time, because nothing in this component ever read a change back. Three of the six landed in this one component, and the other two are further down this page.

Now seed the incidents. Each carries a VECTOR(2048) embedding of its description, produced with jina-embeddings-v4. Because embeddings are large, upsert one row per call (a batched frame carrying several 2048-dim vectors exceeds the gateway's first-frame size). The three resolved incidents (INC1003, INC1004, INC1005) all point at ci_id=1 (pg-orders-primary) and describe the same connection exhaustion in three different wordings — that is the recurrence the vector pillar catches. INC1009 is the fresh one that incident management triaged minutes earlier. INC1007 (a failed reporting extract) is the near-miss the radius must leave out.

You

For each incident, embed its description with jina-embeddings-v4, then upsert the full row (one row per call). Seed the pg-orders-primary recurrence on ci_id 1, service_id svc-orders, assignment_group g-dba, category database: INC1003 (id 1003, P1, resolved, 'The order database ran out of available connections and new sessions were refused; the reporting ETL was holding open cursors', resolved by raising max_connections and killing the leaking reporting-etl cursors); INC1004 (id 1004, P1, resolved, 'Connections to the order database were exhausted a second time; the reporting ETL is again leaking open cursors overnight', resolved by restarting the ETL worker); INC1005 (id 1005, P2, resolved, 'The order database is running out of connections again and the reporting ETL is the suspected source of the leaked sessions', resolved by restarting reporting-etl with a permanent fix still outstanding); INC1009 (id 1009, P1, triaged, 'The order database has run out of connections again and checkout requests are failing; the reporting ETL is suspected once more'). Then the near-miss: INC1007 (id 1007, ci_id 9, service_id svc-reporting, g-orders, availability, P3, new, 'The nightly reporting extract did not complete and no rows were written to the warehouse').

DODIL MCP tools called
ignite_models_embed→data_table_upsert
Agent

Embedded and upserted 5 incidents (each embedding a 2048-dim jina-embeddings-v4 vector, wal_written: true). The four order-database incidents (1003/1004/1005/1009) share ci_id=1 (pg-orders-primary) and g-dba; INC1007 is a reporting failure on a different CI — related in words, not the same problem.

Note service_id is the string "svc-orders", not a number. That is the shape itsm/core owns, and typing it as a BIGINT here is exactly the kind of quiet drift the single-bucket run exists to catch — see incident management for what it cost.

Step 1 — The two tables this skill owns

Problem management adds two SQL tables. problem_incidents is the cluster membership — one merge-keyed link_id row per (problem, incident) pair, so a re-cluster upserts in place and never duplicates a link. known_errors is the KEDB — the published root cause + workaround an on-call searches before touching a recurring symptom.

NOTE

data table create makes every non-PK column NOT NULL by default — set "nullable":true on any optional column (problem_incidents.similarity, known_errors.workaround, known_errors.permanent_fix_change_id), or a partial write 500s with NotNullViolation.

You

In the itsm bucket, create two merge-keyed tables, non-key columns nullable. problem_incidents (key link_id): problem_id(long), incident_id(long), similarity(double), added_at. known_errors (key ke_id): problem_id(long), symptom, workaround, root_cause, permanent_fix_change_id(long), published(boolean), created_at.

DODIL MCP tools called
data_table_create
Agent

Created problem_incidents (key link_id) and known_errors (key ke_id), all optional columns nullable. Upserts are idempotent — re-clustering a problem updates its links in place.

Step 2 — Cluster related incidents (Vector)

Here is the whole reason problem management can be automated: the incidents.embedding column is the recurrence index. Take an incident's symptom, KNN-search the incident history by cosine similarity, and everything inside the cluster_threshold radius (0.25) is the same underlying problem — regardless of wording. Seeded from INC1003, four incidents fall inside the radius across the full 13-incident estate, and there is a visible gap to the fifth:

You

In the itsm bucket, rank every incident that has an embedding by cosine distance from INC1003's embedding, and show which fall inside the 0.25 cluster radius.

DODIL MCP tools called
data_pg
Agent

Inside the 0.25 radius: 1003 (0.000, self), 1004 (0.081), 1009 (0.109), 1005 (0.142) — all ci_id 1 (pg-orders-primary), all assigned to g-dba. The nearest exclusion is 1007 at 0.270 (a failed reporting extract on a different CI), then 1002 (0.308) and the rest of the estate out past 0.37. Four in, and a clean gap to the fifth.

NOTE

Filter WHERE embedding IS NOT NULL. Two of the estate's 13 incidents were created without an embedding, and <=> against a NULL left operand fails the whole query with list_cosine_distance: left argument can not contain NULL values. One un-embedded row takes the cluster scan down for every incident — the same shape as the SLA clock's NULL opened_at outage (see SLA management). A per-tick scan must never be one bad row away from stopping.

Four incidents within the radius clears cluster_min (3), so a problem opens. Two things in that result are worth dwelling on before we write anything.

INC1009 is in the cluster, and this component never triaged it. It was created and classified minutes earlier by incident management — a different component, a different router, its own tutorial. The clusterer sees it because there is no "it": both components read and write the same incidents rows in the same bucket. There is no export, no nightly sync, no vector-store copy to reindex. A ticket triaged at 22:08 is a clustering candidate at 22:09 because it is the same row. That is the single-bucket payoff stated as a fact rather than a diagram.

The threshold is doing real work. INC1007 at 0.270 is about the reporting ETL and shares vocabulary with all four cluster members — it is genuinely close. It is also a different failure on a different CI, and 0.25 correctly leaves it out. A radius that let it in would have merged two problems into one and root-caused neither.

Now write the problem_incidents links (similarity = the cosine distance) and confirm the common CI — the root-cause hint that every member points at the same database. That is a GROUP BY ci_id over the cluster (or a graph lookup if you installed itsm/cmdb-blast-radius):

You

In the itsm bucket, open problem PRB2001 (id 2001) and link its four clustered incidents: upsert problem_incidents rows link_id 2001-1003/2001-1004/2001-1009/2001-1005 (problem_id 2001, the incident_id, the cosine similarity). Then confirm the common CI: group the four incidents by ci_id.

DODIL MCP tools called
data_table_upsert→data_sql
Agent

Upserted 4 problem_incidents links. GROUP BY ci_id over the cluster returns one row: ci_id=1 (pg-orders-primary), count 4 — the whole cluster shares one database, the root-cause hint.

Step 3 — The root-cause synthesis gate (Models)

Clustering decides which incidents belong together; the model decides why. The engine hands kimi-k2.6 the cluster bundle — each incident's symptom, how the earlier ones were fixed, and the common CI — and asks for a strict JSON verdict: root_cause, workaround, is_known_error, and a one-line permanent_fix. The deterministic clustering keeps the model honest; it only synthesises.

IMPORTANT

kimi-k2.6 is a reasoning model — guard the gate against empty content. Through ignite models chat (MCP/CLI) there is no max_tokens knob, so the model can spend its whole budget on hidden reasoning and return an empty content. Two defences, both required: end the prompt with Return ONLY compact JSON, no reasoning/preamble, and retry until the content is non-empty — for kimi-k2.6 a single retry is not enough (this was observed live). Inside the Step 5 handler you also set max_tokens: 4096 on the raw api.dodil.io/v1 call, which the interactive path can't.

You

On kimi-k2.6, synthesise the root cause of a cluster of 4 recurring incidents on CI 1 (pg-orders-primary, a prod database): INC1003 'the order database ran out of available connections and new sessions were refused; the reporting ETL was holding open cursors' fixed by raising max_connections and killing the leaking reporting-etl cursors; INC1004 'connections exhausted a second time, the reporting ETL again leaking open cursors overnight' fixed by restarting the ETL worker; INC1005 'running out of connections again, reporting ETL suspected' fixed by restarting reporting-etl with a permanent fix still outstanding; INC1009 (open) 'run out of connections again and checkout requests are failing, reporting ETL suspected once more'. Return ONLY compact JSON {root_cause, workaround, is_known_error, permanent_fix (<=20 words)}, no reasoning or preamble. Retry until the content is non-empty.

DODIL MCP tools called
ignite_models_chat
Agent

{"root_cause":"Persistent cursor/connection leak in reporting-etl causing resource exhaustion on CI 1.","workaround":"Restart reporting-etl worker, kill leaked cursors, and temporarily raise max_connections.","is_known_error":true,"permanent_fix":"Patch reporting-etl to properly close cursors and release connections after use."}

Read what the model actually did there, because it is the argument for spending a gate call at all. Every individual fix in the cluster was real and correct — raise max_connections, kill the cursors, restart the worker. Each one restored service. None of them was the cause, and the bundle makes that obvious the moment the four are read together: three engineers each treated the symptom on pg-orders-primary, and the leak in reporting-etl was never touched. The permanent fix the model proposes is the first thing anyone has said about reporting-etl itself. Clustering found the recurrence; the gate named the thing nobody had been assigned to fix.

Step 4 — Publish the problem, the known-error, and link a change

The verdict lands as rows. Write the problems record (root_cause, state=known_error, known_error=true, workaround), publish a known_errors row (published=true), stamp incidents.problem_id on all four clustered incidents, and — because link_change is true — spawn a changes row for the permanent fix that back-references the problem, then set problems.related_change_id and known_errors.permanent_fix_change_id to it. Now the recurring outage is one auditable chain: incidents → problem → known-error → change.

You

In the itsm bucket: (1) upsert problem 2001 (number PRB2001, short_description 'Order database connection pool exhausted', the root_cause, state known_error, known_error true, the workaround, related_change_id 3001); (2) upsert known_errors row KE2001 (problem_id 2001, the symptom, workaround, root_cause, permanent_fix_change_id 3001, published true); (3) upsert change 3001 (number CHG3001, ci_id 1, short_description 'Patch reporting-etl to properly close cursors and release connections after use.', type normal, state assess, risk medium, requested_by itsm-problem-management, approval_state pending, problem_id 2001); (4) stamp incidents.problem_id = 2001 on incidents 1003, 1004, 1005, 1009.

DODIL MCP tools called
data_table_upsert→data_table_update
Agent

Wrote problem PRB2001 (known_error), known_errors KE2001 (published), change CHG3001 (problem_id 2001, state assess), and stamped problem_id=2001 on the 4 clustered incidents. The recurrence is now one linked chain.

state: "assess" is not a detail — it is a contract with another component

The obvious value for a brand-new change is state: "new". That is what this engine wrote, it passed every test this component has, and it was completely broken.

Change management's immutability guard only assesses a change from an assessable state — the set {"assess", None}. A change that arrives in state: "new" is rejected:

409  change is frozen in state 'new'

Not once. Forever. There is no transition out of new that the CAB gate offers, because from its point of view new is a state it never issues and does not recognise as a starting point. So the change that a problem record exists in order to deliver — the one fix that would end the recurrence — could never be assessed, never reach the CAB, and never ship. The problem would sit there permanently known_error, with a permanent fix permanently frozen, and every dashboard would show the process working perfectly.

This was invisible for exactly the reason all six were. Standalone, this component's ## Test asserts that a changes row exists with a problem_id back-reference. It does. It never asks change management what it thinks of it, because standalone there is no change management. Both components passed. The seam between them was wrong.

The fix is one word in the producer, and the lesson is worth more than the word: a state machine is a contract between two components, and the producer has to know the consumer's entry state. If you spawn a record another component owns the lifecycle of, the entry state is part of that component's API — as much as the table name or the column types. Read it from the consumer; don't guess a value that "looks like a beginning". The consumer side of this story, including what the CAB does with CHG3001 once it can assess it, is in change management.

Commit before you read back — the write-log is staged

The back-link phase has a second scar in it, and it is a pure DataK3 semantics lesson.

Writing the change is only half of link_change. The engine then has to point problems (related_change_id) and known_errors (permanent_fix_change_id) at it. It does that with the read-merge-upsert idiom used all over this suite: fetch the current row, merge one field, upsert the whole thing back.

Which is where it died:

AttributeError: 'NoneType' object has no attribute 'id'

s.get(Problem, pid) returned None — for a problem row the same handler had written a few lines earlier. DataK3 has no read-your-writes inside an open transaction. The write log is staged until commit, so a SELECT genuinely does not see rows the same transaction just INSERTed. The row was not missing; it was not committed yet.

The fix is a single s.commit() between the write phase and the back-link phase — two transactions, not one:

    # ... upsert problems / problem_incidents / known_errors ...
 
    s.commit()          # <- REQUIRED. The rows above are staged until this line; without it the
                        #    s.get(Problem, pid) below returns None and the merge dies on NoneType.
 
    if link:
        change_id = pid + 1000
        upsert(s, Change, [{... "state": "assess" ...}], key="id")
        prob = s.get(Problem, pid)                       # now visible — committed txn
        prob_row = {c.name: getattr(prob, c.name) for c in Problem.__table__.columns}
        prob_row["related_change_id"] = change_id
        upsert(s, Problem, [prob_row], key="id")

Generalise the shape, because it is not about problems or changes: any handler that writes rows and then re-derives from them must do the re-derive in a second transaction, after the first commits — a ledger balance recomputed from journal lines it just posted, a rollup over rows it just inserted, a back-link like this one. And make the re-derive idempotent while you are there (SUM(...), not +=), so two concurrent runs land the value once rather than twice.

That is three of the six integration bugs in one component: an attribute name that wasn't a column name, a read that couldn't see its own write, and a state the consumer would never accept. Every one of them was found the same way, and not one was findable on a private bucket: components validated separately are internally consistent and still wrong together. Alone, each of the seven ITSM components passed its own ## Test. Stood up on one bucket, as the suite app actually runs, they surfaced six integration bugs in an afternoon. The other three are in core, incident management and SLA management — the last of which is the one where a single NULL column took the SLA clock down for the entire estate.

The whole chain is now one JOIN — the KEDB entry an on-call reads, with its incidents, its root cause, and the change that will end the recurrence:

You

In the itsm bucket, show the problem PRB2001 with its incident count, its known-error workaround, and the linked change: join problems to problem_incidents, known_errors, and changes.

DODIL MCP tools called
data_sql
Agent

One row: PRB2001, 4 incidents, known_error true, KE2001 published with a workaround, change CHG3001 (assess, pending) — the recurrence as a single auditable record.

From here CHG3001 is change management's problem, literally. It goes to the CAB carrying the blast radius of CI 1 — pg-orders-primary is the hub of this estate, and the assessment comes back cab_review across 4 impacted business services rather than an auto-approval. That is the right answer: a permanent fix to the database everything depends on is exactly the change a human should look at. The mechanics are in change management.

Routes

The steps above are the engine's inner loop by CLI. The download (see Get the code) fronts the same bucket with a small FastAPI app, routes.py — incident/problem CRUD plus the three ops that turn recurring incidents into an auditable problem. This is what you deploy; every route obeys the same DataK3 rules the package bakes in, so quoting it is documenting them.

The connection and the one write helper live in db.py — byte-identical to the whole suite's db.py. A DataK3 bucket is a Postgres endpoint (db name = the bucket, user = the literal token, password = your DODIL token), so there's no data connect step in code, just fixed region constants. upsert() is the only writer every route uses — INSERT … ON CONFLICT (pk) DO UPDATE:

# db.py — the idempotent keyed write every route uses (INSERT ... ON CONFLICT DO UPDATE)
def upsert(session, model, rows, key):
    keys = [key] if isinstance(key, str) else list(key)
    table = model.__table__
    cols = {c for r in rows for c in r}
    rows = [{c: r.get(c) for c in cols} for r in rows]
    stmt = pg_insert(table).values(rows)
    update_cols = [c.name for c in table.columns if c.name not in keys]
    if update_cols:
        stmt = stmt.on_conflict_do_update(
            index_elements=keys,
            set_={c: getattr(stmt.excluded, c) for c in update_cols},
        )
    else:
        stmt = stmt.on_conflict_do_nothing(index_elements=keys)
    session.execute(stmt)

Why it matters: on DataK3 a bare re-INSERT of an already-committed primary key raises duplicate-key 23505 — a plain INSERT is not an upsert on re-write. upsert makes a re-cluster, a shard replay, or a re-import land the row once (proven live below: re-upserting the three links keeps count(*) == count(DISTINCT link_id) == 3).

CRUD — each write is an upsert then a commit (DataK3 has no read-your-writes inside an open transaction; the engine is expire_on_commit=False, so routes commit before they return):

# routes.py — upsert an incident (embedding included), keyed on the natural PK
@app.post("/incidents")
def upsert_incident(i: IncidentIn, s: Session = Depends(db)):
    """Upsert an incident (embedding included). Keyed on `id`, so a replayed webhook or a
    re-import lands the row once."""
    row = i.model_dump()
    row["opened_at"] = _now()
    upsert(s, Incident, [row], key="id")
    s.commit()
    return {"ok": True, "id": i.id}

Workflow op 1 — cluster → root-cause → publish (VECTOR + MODELS). POST /incidents/{incident_id}/cluster is the whole inner loop as one route. It KNN-scans incidents.embedding from the seed (Incident.embedding.cosine_distance(seed.embedding), filtered <= CLUSTER_THRESHOLD — the exact <=> query proven live: from INC1003 four incidents fall at 0.000 / 0.081 / 0.109 / 0.142, inside the 0.25 radius, while the nearest non-member sits at 0.270). When >= CLUSTER_MIN fall inside it opens a problems row, writes the problem_incidents links, stamps incidents.problem_id, and — if auto_known_error — calls the kimi-k2.6 gate and publishes a known_errors row; if link_change, spawns the permanent-fix changes row. The problem id is idempotent: it reuses the seed's problem_id if the seed is already clustered, else max(id)+1, so a re-cluster lands the same links + the same problem:

# routes.py — the money step: VECTOR KNN + the Models gate + the linked writes (excerpt)
@router.post("/incidents/{incident_id}/cluster")
def cluster(incident_id: int, q: ClusterIn = ClusterIn(), user=Depends(current_user),
            s: Session = Depends(db)):
    auto = AUTO_KNOWN_ERROR if q.auto_known_error is None else q.auto_known_error
    link = LINK_CHANGE if q.link_change is None else q.link_change
    seed = s.get(Incident, incident_id)
    # VECTOR KNN — the cluster is every incident within CLUSTER_THRESHOLD by cosine distance
    members = s.execute(
        select(Incident.id, Incident.ci_id,
               Incident.embedding.cosine_distance(seed.embedding).label("d"),
               Incident.resolution)
        .where(Incident.embedding.cosine_distance(seed.embedding) <= CLUSTER_THRESHOLD)
        .order_by("d")
    ).all()
    if len(members) < CLUSTER_MIN:
        return {"clustered": False, "size": len(members)}
    # idempotent problem id: reuse the seed's if already clustered, else max(id)+1 (from 2001)
    pid = seed.problem_id or (s.execute(
        text("SELECT COALESCE(MAX(id), 2000) + 1 FROM problems")).scalar())
    verdict = _rca(_models_token(), bundle) if auto else {}
    # ... upsert problems / problem_incidents / known_errors ...
    s.commit()                     # REQUIRED before the back-link phase reads any of it
    if link:
        # the spawned change enters in state "assess" — the state change management accepts
        ...

Note the route takes Depends(current_user) and nothing more. That is deliberate, and it is a change from how this post used to read — see Auth below.

The gate (_rca) is the reasoning-model guard, verbatim from the package: max_tokens: 4096, the response wrapped in data, and retry until the content is non-empty (a single retry is not enough for kimi-k2.6). It mints a service-account client_credentials token and sets an explicit User-Agent (the edge 403s the stdlib default):

# routes.py — the root-cause gate (kimi-k2.6): retry UNTIL non-empty (reasoning-model guard)
def _rca(token: str, bundle: str) -> dict:
    client = httpx.Client(base_url=MODELS_BASE, headers={**_UA, "Authorization": f"Bearer {token}"})
    body = {"model": CHAT_MODEL, "max_tokens": 4096,
            "messages": [{"role": "system", "content": SYS},
                         {"role": "user", "content": bundle}]}
    for _ in range(5):                               # retry until non-empty
        out = client.post("/chat/completions", json=body, timeout=120).json()
        env = out.get("data", out)                   # this platform wraps the response in `data`
        content = (env["choices"][0]["message"]["content"] or "").strip()
        if content:
            m = re.search(r"\{.*\}", content, re.DOTALL)
            return json.loads(m.group(0) if m else content)
    raise HTTPException(502, "RCA gate returned empty content after retries")

Workflow op 2 — the recurrence signal alone (VECTOR). POST /incidents/similar takes a query embedding and returns the nearest incidents by pgvector cosine distance — the cluster signal on its own, callable before you decide to open a problem:

# routes.py — nearest incidents to a symptom embedding (VECTOR)
@router.post("/incidents/similar")
def similar_incidents(q: SimilarIn, user=Depends(current_user), s: Session = Depends(db)):
    rows = s.execute(
        select(Incident.id, Incident.embedding.cosine_distance(q.embedding).label("d"))
        .order_by("d")
        .limit(q.top_k)
    ).all()
    return {"matches": [{"incident_id": iid, "distance": float(dist)} for iid, dist in rows]}

Workflow op 3 — the linked chain (SQL). GET /problems/{problem_id}/chain is the on-call's one-JOIN answer — the problem, its incident count, its published known-error, and the linked change. Live-verified on the itsm estate it returns PRB2001 | known_error | 4 | KE2001 | true | CHG3001 | pending:

# routes.py — the recurrence as one auditable record (SQL JOIN)
@router.get("/problems/{problem_id}/chain")
def problem_chain(problem_id: int, user=Depends(current_user), s: Session = Depends(db)):
    row = s.execute(text(
        "SELECT p.number AS problem, p.state, "
        "(SELECT count(*) FROM problem_incidents pi WHERE pi.problem_id = p.id) AS incidents, "
        "k.ke_id, k.published, c.number AS change_number, c.approval_state "
        "FROM problems p "
        "LEFT JOIN known_errors k ON k.problem_id = p.id "
        "LEFT JOIN changes c ON c.id = p.related_change_id "
        "WHERE p.id = :pid"), {"pid": problem_id}).mappings().first()
    if not row:
        raise HTTPException(404, "no such problem")
    return dict(row)

Adding a new operation touches only routes.py (and maybe models.py) — the plumbing in db.py is fixed. The pattern is one Pydantic *In schema + one @router.<verb> function: write via upsert, vector via cosine_distance(…), the gate via the retry-until-non-empty _rca (see EXTENDING.md).

Auth — config at the edge, a role gate in the app

On Ignite, end-user login is configuration, not code. The ITSM deploys with the itsm-suite dodil-appid pool attached (user_pool: itsm-suite in .dodil/deploy.yaml) and the per-cluster Ignite gateway runs the entire browser login at the edge — PKCE S256 against the pool issuer, an AEAD-sealed host-only session cookie, single-flight refresh, EdDSA JWT verification against trust anchors this app does not hold — then injects the verified identity into every request it forwards: X-Dodil-User (sub, email, connection, app_roles), X-Dodil-User-Jwt (the raw verified token, carrying the catalog-expanded permissions claim) and X-Dodil-Auth-Source (pool for an app end-user, platform for an operator or service-account invoke). Any inbound copy of those headers is stripped first, on every mode and every principal, so a caller can never forge them.

What survives in the package is a small auth.py that ships no verifier — no JWKS client, no issuer/audience env, no crypto dependency, and no pyjwt in requirements.txt. It reads the injected header and keeps the one job the app still owns: role-based gating.

# auth.py — a header-trust reader, not a verifier (the gateway already checked the JWT)
def current_user(request: Request) -> dict:
    raw = request.headers.get("x-dodil-user")   # {"sub","email","connection","app_roles"}
    ...                                          # + permissions read off x-dodil-user-jwt
    raise HTTPException(401, "end-user login required — no X-Dodil-User from the gateway")
 
def require_permission(perm: str):
    """Gate a route on a pool permission: Depends(require_permission("itsm:change:approve"))."""
    def _dep(user: dict = Depends(current_user)) -> dict:
        if perm not in user["permissions"] and perm not in user["roles"]:
            raise HTTPException(403, f"missing permission: {perm}")
        return user
    return _dep

ITSM uses namespaced permissions — <module>:<object>:<verb> — so one customer pool can carry every ERP module's roles without collision (itsm:change:approve is not crm:change:approve). Across all seven components the audit left exactly four gates standing:

PermissionGatesWhy this one and not the rest
itsm:change:approvechange-management: POST /changes/{id}/assess, POST /changes/{id}/transition, POST /change-policiesaccepting the risk of a production change. change_approvals.decided_by records the gateway-vouched user
itsm:major:declaremajor-incident: POST /major/declaredeclaring a major incident pages the org. bridges.declared_by records who
itsm:incident:resolvemajor-incident: POST /major/bridges/{id}/statethe only route in the module that writes incidents.state='resolved' — what stops the SLA clock
itsm:cmdb:rebuildcmdb-blast-radius: POST /graph/assembleit TRUNCATEs impacts + service_map and DROP+CREATEs both graphs; every other component's blast radius is computed from what it leaves behind

Problem management has none of them, and that is the interesting part of this section.

An earlier version of this post gated POST /incidents/{id}/cluster on a problems:write permission, so that only a problem_manager role could open a problem record. The audit deleted that gate, along with incidents:triage and cmdb:write in the neighbouring components. The argument is worth making properly, because "add a permission" always feels like the responsible choice:

  • Clustering writes nothing a human can't undo, and accepts no risk. It groups incidents that are already in the bucket, records a root-cause hypothesis, and files a change for someone else to approve. The gate that matters — the one where a person accepts the risk of touching production — is itsm:change:approve, and it lives one component downstream, on the CAB. That gate is real precisely because this one isn't.
  • Analysis is the service desk's ordinary work. Asking "have we seen this before, and why does it keep happening?" is the job. A permission that every agent on the desk must hold in order to do their job is not a control; it is a checkbox with a support ticket attached, and the first time it blocks someone at 3am it gets granted to everyone permanently.
  • A permission nobody is ever denied is worse than no permission, because it looks like a control in an audit. An auditor discounts ceremony — and correctly, since a gate held by all is evidence of nothing. Four real gates that a specific, small set of people hold is a defensible answer to "who can accept production risk here?". Seven gates, three of which everyone holds, is not.

Note also that itsm:cmdb:rebuild was added by the audit — a fourth permission the original design had not predicted — because the route it guards destroys and rebuilds the graph every other component reads. The audit did not simply remove gates; it moved them to where the irreversible things happen.

So in this component every route takes Depends(current_user) and nothing more: a signed-in service desk user may cluster, may open a problem, may publish a known error, and may file the fix change. The decided_by on that change's eventual approval will name someone else.

The pool is created once for the whole suite, with the role catalog those four gates check:

You

Create a dodil-appid pool itsm-suite with email+password, and set its role catalog: the service desk agent role holds no special permissions; a change manager may approve changes; an incident commander may declare a major incident and resolve one; a CMDB admin may rebuild the graph.

Agent

Pool itsm-suite created — issuer https://appid.dodil.io/ihdiash/itsm-suite, audience pool:itsm-suite, email+password (local) enabled. Catalog set: agent = (no permissions); change-manager = itsm:change:approve; incident-commander = itsm:major:declare, itsm:incident:resolve; cmdb-admin = itsm:cmdb:rebuild. A user's next token carries app_roles plus the catalog-expanded permissions claim the gates read.

agent — the service desk, and the largest role by far — holds no permissions at all. It does the ungated work, which is most of the module: all of this component, all of incident management, all of SLA management, and CMDB CRUD.

The two-plane rule is unchanged: the pool identifies the user; the app still reaches DataK3 through its own service account (sa_token.py mints and refreshes a client_credentials token for the pg-wire password) — an app-user is never a bucket principal. Locally, with no gateway in front of uvicorn routes:app, opt in to a stub identity with DEV_ALLOW_ANON=1; the stub carries no permissions unless you grant them (DEV_USER_PERMISSIONS=itsm:change:approve,…), so the gated routes stay gated on a laptop too. The full flow — creating the pool, the redirect_uris allowlist, what the gateway injects, and the off-gateway path where you do verify the pool JWT yourself — is App authentication; the catalog mechanics are App roles. Today the pool is email+password (local); oauth/oidc/saml corporate SSO switch on per pool later, no app change.

The route with no identity at all

There is one more case, and it is the one that generalises furthest. POST /sla/tick — the SLA clock in the neighbouring component — carries no identity dependency whatsoever: not a permission, not even Depends(current_user). It is a machine heartbeat, called by a service account over a platform invoke, which carries no X-Dodil-User at all. A current_user dependency there would 401 the clock; the engine would stop, and every incident's breach flags would silently go stale — the failure mode of an SLA system that reports green because it has stopped counting. Exposure is the ingress's job (public_invoke=false), not a user permission.

This applies to every recompute route in every module — a rollup, a re-materialization, an embedding backfill, this component's own clusterer tick. If a machine calls it, gate the door, not the caller.

Step 5 — The clustering engine on Ignite (itsm-problem-clusterer)

The steps above are the engine's inner loop. In production one Ignite app, itsm-problem-clusterer, does it on a tick: KNN-scan the open incidents, group the tight clusters, open the problem, call the RCA gate, publish the known-error, and spawn the change — all over one Postgres-wire connection (pgvector <=> KNN, the cluster GROUP BY, and every write on the same psycopg cursor). It's a separate workload, so it gets its own service account, and because it both writes DataK3 and calls Models it needs three live roles (confirm the exact names with dodil auth service-account list-roles — done live 2026-09-02):

  • k3.editor — write problems, problem_incidents, known_errors, changes.
  • ignite.model-user — call kimi-k2.6 from inside the handler (token-billed).
  • ignite.app-developer — the deploy/invoke identity for the app itself.

NOTE

There is no ignite.developer role (an older doc named one) — the live catalog splits it into ignite.app-developer (deploy/invoke) and ignite.model-user (call models).

This ships as an image-mode Ignite app: a plain HTTP server (GET /healthz for the probe, POST /cluster for the work), packaged by a Dockerfile and built on deploy — not a handler(payload, ctx) compile-mode function. Three things matter and each is a line below:

  • The Models call is the real OpenAI-compatible endpoint (api.dodil.io/v1), authed with a service-account token, max_tokens: 4096 (kimi-k2.6 is a reasoning model — a low budget returns empty content), the response wrapped in data, and retried until the content is non-empty.
  • The writes go over the drop-in Postgres wire (pg.uk-lon-1.dodil.io:5432, dbname=$BUCKET, user=token, password=$SA_ACCESS_TOKEN) via psycopg — there is no K3 HTTP API. A first cluster mints a fresh problem id (max(id)+1), so those are append-only first-writes. Re-clustering the same seed is not — it reuses the seed's problem_id, and the change id is deterministic (pid + 1000) — so any re-run path must write INSERT … ON CONFLICT (<pk>) DO UPDATE SET col = EXCLUDED.col. A bare re-INSERT of an already-committed PK raises duplicate-key 23505; a plain INSERT is not an upsert on re-write. (The FastAPI package sidesteps this entirely — every write goes through db.upsert.) Writes retry on SerializationFailure.
  • Every call to id.dodil.io / api.dodil.io sets an explicit User-Agent — stdlib urllib's default is Cloudflare-banned (HTTP 403 "error code: 1010").
# server.py — itsm-problem-clusterer, an IMAGE-mode Ignite app (HTTP server on $PORT).
#   GET  /healthz  -> {"status":"ready"}   (probe; no auth)
#   POST /cluster  -> {"incident_id": <seed>} -> cluster, RCA-gate, publish problem/KEDB/change
 
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
from psycopg import errors as pg_errors
 
# --- hoisted knobs: mirror the skill params, injected as env at deploy ---
CLUSTER_MIN       = int(os.environ.get("CLUSTER_MIN", "3"))
CLUSTER_THRESHOLD = float(os.environ.get("CLUSTER_THRESHOLD", "0.25"))
AUTO_KNOWN_ERROR  = os.environ.get("AUTO_KNOWN_ERROR", "true").lower() == "true"
LINK_CHANGE       = os.environ.get("LINK_CHANGE", "true").lower() == "true"
EMBED_MODEL       = os.environ.get("EMBED_MODEL", "jina-embeddings-v4")
CHAT_MODEL        = os.environ.get("MODEL_ID", "kimi-k2.6")
BUCKET            = os.environ["BUCKET"]
SA_ID             = os.environ["DODIL_SERVICE_ACCOUNT_ID"]   # the cli-… serviceAccountId, NOT the uuid
SA_SECRET         = os.environ["DODIL_SERVICE_ACCOUNT_SECRET"]
PG_HOST           = os.environ.get("PG_HOST", "pg.uk-lon-1.dodil.io")
PG_PORT           = int(os.environ.get("PG_PORT", "5432"))
 
ID_URL     = "https://id.dodil.io/realms/dodil/protocol/openid-connect/token"
EMBED_URL  = "https://api.dodil.io/v1/embeddings"
CHAT_URL   = "https://api.dodil.io/v1/chat/completions"
UA         = "itsm-problem-clusterer/1.0"   # explicit UA — stdlib urllib default is Cloudflare-banned (403 1010)
 
SYS = ("You are an ITSM problem-management assistant. Given a cluster of related incidents "
       "(symptoms, the common CI, and how earlier ones were fixed), synthesise the underlying "
       "root cause. Return ONLY compact JSON, no reasoning or preamble: "
       '{"root_cause": string, "workaround": string, "is_known_error": boolean, '
       '"permanent_fix": string (<=20 words)}.')
 
def _now(): return datetime.now(timezone.utc).isoformat()
 
def _http_post(url, data, headers, form=False):
    headers = {"User-Agent": UA, **headers}          # the UA is required (see above)
    if form:
        body = urllib.parse.urlencode(data).encode()
        headers["Content-Type"] = "application/x-www-form-urlencoded"
    else:
        body = json.dumps(data).encode()
        headers["Content-Type"] = "application/json"
    req = urllib.request.Request(url, data=body, headers=headers, method="POST")
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.loads(r.read().decode())
 
def _token():                                        # OIDC client_credentials -> access token
    out = _http_post(ID_URL, {"grant_type": "client_credentials",
                              "client_id": SA_ID, "client_secret": SA_SECRET},
                     headers={}, form=True)
    return out["access_token"]
 
def _embed(token, text):                             # jina-embeddings-v4 -> a pgvector literal
    out = _http_post(EMBED_URL, {"model": EMBED_MODEL, "input": text},
                     headers={"Authorization": f"Bearer {token}"})
    vec = (out.get("data", out))["data"][0]["embedding"]
    return "[" + ",".join(repr(round(x, 6)) for x in vec) + "]"
 
def _rca(token, bundle):                             # kimi-k2.6: max_tokens 4096, retry UNTIL non-empty
    for _ in range(5):
        out = _http_post(CHAT_URL,
                         {"model": CHAT_MODEL, "max_tokens": 4096,
                          "messages": [{"role": "system", "content": SYS},
                                       {"role": "user", "content": bundle}]},
                         headers={"Authorization": f"Bearer {token}"})
        env = out.get("data", out)                   # response wrapped in "data" on this platform
        content = (env["choices"][0]["message"]["content"] or "").strip()
        if content:
            return _extract_json(content)
        time.sleep(1)                                # reasoning burned the budget — try again
    raise ValueError("RCA gate returned empty content after retries")
 
def _pg(token):
    return psycopg.connect(host=PG_HOST, port=PG_PORT, dbname=BUCKET,
                           user="token", password=token, sslmode="require",
                           connect_timeout=20, autocommit=False)
 
def _extract_json(text):
    text = text.strip()
    if text.startswith("```"):
        text = re.sub(r"^```[a-zA-Z]*\n?", "", text)
        text = re.sub(r"\n?```$", "", text).strip()
    m = re.search(r"\{.*\}", text, re.DOTALL)
    return json.loads(m.group(0) if m else text)
 
def _retry(fn):                                      # pg engine is serializable — retry transient conflicts
    for attempt in range(4):
        try:
            return fn()
        except (pg_errors.SerializationFailure, pg_errors.DeadlockDetected):
            if attempt == 3: raise
            time.sleep(0.4 * (attempt + 1))
 
def cluster_from(seed_id):
    token = _token()
    with _pg(token) as conn, conn.cursor() as cur:
        # the seed incident + its embedding
        cur.execute("SELECT description, ci_id, embedding FROM incidents WHERE id = %s", (seed_id,))
        desc, ci_id, qvec = cur.fetchone()
        # VECTOR KNN — the cluster is every incident within CLUSTER_THRESHOLD by cosine distance
        cur.execute(
            "SELECT id, ci_id, embedding <=> %s AS dist, resolution "
            "FROM incidents WHERE (embedding <=> %s) <= %s ORDER BY dist",
            (qvec, qvec, CLUSTER_THRESHOLD))
        members = cur.fetchall()
        if len(members) < CLUSTER_MIN:
            return {"clustered": False, "size": len(members)}   # not a recurrence yet
 
        # a new problem id (max+1), then the links + the common-CI hint
        cur.execute("SELECT coalesce(max(id), 2000) + 1 FROM problems")
        pid = cur.fetchone()[0]
        common_cis = {m[1] for m in members}
        bundle = "\n".join(f"- incident {m[0]} (ci {m[1]}); prior fix: {m[3] or 'open'}" for m in members)
        bundle = f"Common CI: {ci_id if len(common_cis) == 1 else 'mixed'}\n{bundle}"
 
        verdict = _rca(token, bundle) if AUTO_KNOWN_ERROR else {}
        change_id = None
        def _writes():
            with _pg(token) as w, w.cursor() as c:
                c.execute("INSERT INTO problems (id, number, short_description, root_cause, state, "
                          "known_error, workaround, created_at) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
                          (pid, f"PRB{pid}", desc[:120], verdict.get("root_cause"),
                           "known_error" if AUTO_KNOWN_ERROR else "analysis",
                           bool(verdict.get("is_known_error")), verdict.get("workaround"), _now()))
                for m in members:
                    c.execute("INSERT INTO problem_incidents (link_id, problem_id, incident_id, "
                              "similarity, added_at) VALUES (%s,%s,%s,%s,%s)",
                              (f"{pid}-{m[0]}", pid, m[0], float(m[2]), _now()))
                    c.execute("UPDATE incidents SET problem_id = %s WHERE id = %s", (pid, m[0]))
                if AUTO_KNOWN_ERROR:
                    c.execute("INSERT INTO known_errors (ke_id, problem_id, symptom, workaround, "
                              "root_cause, published, created_at) VALUES (%s,%s,%s,%s,%s,%s,%s)",
                              (f"KE{pid}", pid, desc[:200], verdict.get("workaround"),
                               verdict.get("root_cause"), True, _now()))
                w.commit()
        _retry(_writes)
 
        if LINK_CHANGE:
            change_id = pid + 1000
            def _chg():
                with _pg(token) as w, w.cursor() as c:
                    # state "assess", NOT "new": change management only assesses from an
                    # assessable state ({"assess", None}). A change spawned as "new" is
                    # rejected 409 "frozen in state 'new'" forever — the permanent fix could
                    # never clear the CAB. The producer has to know the consumer's entry state.
                    c.execute("INSERT INTO changes (id, number, ci_id, short_description, type, state, "
                              "risk, requested_by, approval_state, problem_id) "
                              "VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)",
                              (change_id, f"CHG{change_id}", ci_id,
                               verdict.get("permanent_fix", "Permanent fix")[:200],
                               "normal", "assess", "medium", "itsm-problem-clusterer", "pending", pid))
                    c.execute("UPDATE problems SET related_change_id = %s WHERE id = %s", (change_id, pid))
                    c.execute("UPDATE known_errors SET permanent_fix_change_id = %s WHERE problem_id = %s",
                              (change_id, pid))
                    w.commit()
            _retry(_chg)
 
    return {"clustered": True, "problem_id": pid, "size": len(members),
            "incidents": [m[0] for m in members], "verdict": verdict, "change_id": change_id}
 
class Handler(BaseHTTPRequestHandler):
    def _send(self, code, body):
        payload = json.dumps(body).encode()
        self.send_response(code); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload))); self.end_headers()
        self.wfile.write(payload)
    def do_GET(self):
        if self.path == "/healthz": return self._send(200, {"status": "ready"})
        return self._send(404, {"error": "no_route", "path": self.path})
    def do_POST(self):
        if self.path != "/cluster": return self._send(404, {"error": "no_route", "path": self.path})
        try:
            n = int(self.headers.get("Content-Length") or 0)
            body = json.loads(self.rfile.read(n) or b"{}")
            if not body.get("incident_id"): return self._send(400, {"error": "missing incident_id"})
            return self._send(200, cluster_from(int(body["incident_id"])))
        except urllib.error.HTTPError as e:
            return self._send(502, {"error": "upstream", "code": e.code,
                                    "body": e.read().decode(errors="replace")[:600]})
        except Exception as e:
            return self._send(500, {"error": type(e).__name__, "detail": str(e)[:600]})
    def log_message(self, *a): pass
 
if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8080"))
    print(f"itsm-problem-clusterer on 0.0.0.0:{port} bucket={BUCKET} "
          f"min={CLUSTER_MIN} thr={CLUSTER_THRESHOLD}", flush=True)
    ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()

Only psycopg is a third-party dep; everything else is stdlib. The two sibling files that make it an image — ./clusterer/Dockerfile and ./clusterer/requirements.txt:

# ./clusterer/Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY server.py .
ENV PORT=8080
EXPOSE 8080
CMD ["python", "server.py"]
# ./clusterer/requirements.txt
psycopg[binary]==3.2.3

Give it a least-privilege identity, then deploy.

IMPORTANT

Ignite is request-invoked, and there is no server-side scheduler. This is the platform gap every ITSM deployment hits, so decide it deliberately rather than discovering it. A clusterer that only runs when somebody clicks a button is a report, not an engine — and the same argument applies with far more force to the SLA clock next door, where "it only advances when someone loads a page" means breach flags that are quietly wrong. There are two real options and a deployment has to pick one and say which:

  1. An always-on pinned app — deploy with --reserved 1 --max-replicas 1 and run the poll loop inside the pod, calling the work on an interval. Self-contained and nothing outside DODIL has to stay healthy; the cost is a warm replica that never scales to zero.
  2. An external scheduler — cron, a CI timer, any orchestrator that POSTs the route on a schedule. The app stays scale-to-zero and cheap; the cost is that your clock now depends on infrastructure outside the platform, and an outage there is silent.

Either way the caller is a service account over a platform invoke, not a browser user — which is exactly why a recompute route must not sit behind current_user (see Auth above). The full version of this argument, and what goes wrong when nobody makes the choice, is in SLA management.

Run the clusterer on your own tick (POST /cluster with a seed incident), or as a pinned warm poll loop:

You

Create a service account itsm-problem-clusterer-sa, grant it k3.editor plus ignite.model-user and ignite.app-developer, then deploy my ./clusterer app (image mode — its Dockerfile builds on deploy) to Ignite as itsm-problem-clusterer on port 8080 with health path /healthz, passing the service-account creds and the knobs (CLUSTER_MIN 3, CLUSTER_THRESHOLD 0.25, AUTO_KNOWN_ERROR true, LINK_CHANGE true) as runtime env. Use the cli- serviceAccountId as DODIL_SERVICE_ACCOUNT_ID, not the uuid. Then POST a seed incident to /cluster to smoke-test.

DODIL MCP tools called
auth_service-account_create→auth_service-account_grant-role→ignite_app_deploy
Agent

Created itsm-problem-clusterer-sa (serviceAccountId cli-itsm-problem-clusterer-sa), granted k3.editor + ignite.model-user + ignite.app-developer, built + deployed itsm-problem-clusterer (image build, public FQDN on :8080, scale-to-zero). POST /cluster {incident_id:1003} returned clustered=true, problem_id 2001, size 4, change_id 3001.

NOTE

Deploy: image mode (Lane B). The deploy + /cluster smoke-test above use image mode — a Dockerfile + --dockerfile-path, Kaniko build-on-deploy — the identical handler/deploy pattern the CRM reference engine (crm-lead-scorer) proved end-to-end live 2026-09-02 (deploys, serves /healthz + its route unauthenticated, writes durably). This post's clustering, the RCA gate, the pg-wire writes, and the re-cluster are each proven one call at a time in Steps 2–4 and ## Test. If dodil ignite app deploy returns IAM resource registration failed: broken pipe, deploy under a fresh app name — the half-created app can't be updated or deleted.

How the pillars map

One bucket, and the problem-management engine reaches across it — no second system to sync.

JobThe usual stackOn DataK3
Find the recurrenceManual triage / string searchVector — cosine KNN over core incidents.embedding, catches the same problem in different wordings
Cluster membershipA CRM custom object + app codeproblem_incidents — merge-keyed SQL links, re-clustered in place
The known-error / KEDBA wiki page nobody updatesknown_errors — a queryable SQL row, published flag, JOINed to its incidents
Root-cause synthesisA human writing it up days laterkimi-k2.6 on Ignite Models — one auth context, token-billed
Link the permanent fixA ticket copied by hand into the change toola changes row with problem_id back-reference — same bucket, one JOIN

Because the incidents, the problem, the known-error, and the change are all in one bucket, the on-call's "is this a known problem, and what's the workaround?" is a single JOIN over one copy of the rows.

Customize — the decisions this skill asks you

Q1 · cluster_min — how many recurrences before a problem opens?

"How many similar incidents before you open a problem?" → Default 3. A genuine recurrence, not a one-off. Lower opens more problems (noisier, more false patterns); higher waits for a clearer signal before spending an RCA gate call.

Q2 · cluster_threshold — how tight is "the same problem"?

"How tight is 'the same underlying problem'?" → Default 0.25 cosine radius. On the live estate the four pg-orders-primary connection incidents (0.000 / 0.081 / 0.109 / 0.142) cluster, while the failed reporting extract (0.270) and everything else (0.308 and out) stay out. Note how little headroom there is between the last member at 0.142 and the first exclusion at 0.270 — that gap is the setting. Tighten toward 0.15 for near-identical only; loosen for a broader net, and watch INC1007 join a problem it does not belong to.

Q3 · auto_known_error — write the KEDB, or just cluster?

"Should the model write the known-error record, or just cluster?"

  • true (default) → the kimi-k2.6 gate synthesises root_cause + workaround and publishes a known_errors row; incidents.problem_id is stamped.
  • false → clusters only (problem_incidents), leaving RCA and the KEDB entry to a human. The service account then needs no ignite.model-user.

Q4 · link_change — spawn the permanent-fix change?

"Spawn a change for the permanent fix?"

  • true (default) → upserts a changes row (with problem_id back-reference) and sets problems.related_change_id — hands the fix to change management.
  • false → stops at the known-error; no change is created.

Industry overlays (finserv / manufacturing) compose this skill with policy tweaks — e.g. a regulated CI's known-error requires an approver sign-off before published. See the per-industry ITSM pages.

Test

Every command below ran live against DataK3 on 2026-09-08 (org IHDIASH, bucket itsm), one call at a time — and, unlike earlier runs of this post, on the same bucket as the other six ITSM components, with their tables and their rows present. That is what surfaced the three bugs above. Both tested_branches are covered: {cluster_min: 3, auto_known_error: true, link_change: true} (the full path — one live RCA gate call) and {link_change: false} (cluster + KEDB, no change). The Ignite deploy in Step 5 uses the image-mode pattern proven live on crm-lead-scorer; the clustering, the gate, and every write are proven here.

# 1. VECTOR clustering — 4 incidents inside the 0.25 radius; the nearest exclusion is 0.270
dodil data pg -b "$BUCKET" "
  SELECT i.id, ROUND((i.embedding <=> (SELECT embedding FROM incidents WHERE id=1003))::numeric,4) AS dist
  FROM incidents i WHERE i.embedding IS NOT NULL ORDER BY dist"
#  1003 0.0000 | 1004 0.0810 | 1009 0.1090 | 1005 0.1420 || 1007 0.2700 | 1002 0.3080 | ... 0.4740
 
# 2. the 4 incidents cluster into ONE problem; problem_incidents has 4 rows
dodil data sql -b "$BUCKET" "SELECT problem_id, count(*) AS n FROM problem_incidents WHERE problem_id=2001 GROUP BY problem_id"
#  2001 | 4
 
# 3. the RCA gate returned valid JSON {root_cause, workaround, is_known_error, permanent_fix}; is_known_error=true (Step 3)
dodil data sql -b "$BUCKET" "SELECT number, state, known_error, root_cause FROM problems WHERE id=2001"
#  PRB2001 | known_error | true | Persistent cursor/connection leak in reporting-etl causing resource exhaustion on CI 1.
 
# 4. known-error published with a non-empty workaround
dodil data sql -b "$BUCKET" "SELECT ke_id, published, (workaround IS NOT NULL) AS has_workaround FROM known_errors WHERE problem_id=2001"
#  KE2001 | true | true
 
# 5. link_change=true -> a change exists with problem_id back-reference; problems.related_change_id set.
#    state='assess' is the assertion that matters: 'new' would be frozen 409 at the CAB forever.
dodil data sql -b "$BUCKET" "
  SELECT c.number AS change_number, c.state, c.problem_id, p.related_change_id
  FROM changes c JOIN problems p ON p.id = c.problem_id WHERE c.problem_id=2001"
#  CHG3001 | assess | 2001 | 3001
 
# 6. common-CI hint — every clustered incident shares ci_id=1 (pg-orders-primary)
dodil data sql -b "$BUCKET" "SELECT count(DISTINCT ci_id) AS distinct_cis, min(ci_id) AS ci FROM incidents WHERE problem_id=2001"
#  distinct_cis=1 | ci=1
 
# 7. idempotent recluster — re-upsert the 4 links, count stays 4 (one row per link, no duplicates)
dodil data sql -b "$BUCKET" "SELECT count(*) AS total, count(DISTINCT link_id) AS distinct_links FROM problem_incidents"
#  total=4 | distinct_links=4
 
# 8. the cross-component assertion — INC1009 was created and triaged by incident management, and this
#    component clustered it, with no export or sync between them. Same rows, same bucket.
dodil data sql -b "$BUCKET" "SELECT id, number, state, problem_id FROM incidents WHERE id=1009"
#  1009 | INC1009 | triaged | 2001

One-shot

With the DODIL MCP connected, paste this to build the whole clusterer at once on your ITSM bucket:

On my DataK3 bucket `itsm` (which already has incidents with an embedding VECTOR(2048) column, plus
problems and changes), build a problem-management engine. Confirm each step.
 
1. Create merge-keyed tables: problem_incidents (key link_id: problem_id long, incident_id long,
   similarity double, added_at) and known_errors (key ke_id: problem_id long, symptom, workaround,
   root_cause, permanent_fix_change_id long, published boolean, created_at). All non-key columns nullable.
   In the ORM models, every attribute name must equal its column name — never map `type` to `change_type`.
2. Cluster: KNN over incidents.embedding (jina-embeddings-v4, cosine) from a seed incident's symptom,
   filtering `WHERE embedding IS NOT NULL` (one un-embedded row fails the whole scan); incidents within
   cluster_threshold 0.25 form the cluster. When >= cluster_min (3) fall inside AND share a CI, open a
   problems row and write problem_incidents links (similarity = cosine distance).
3. RCA gate: give kimi-k2.6 the cluster (each incident's symptom + how earlier ones were fixed + the
   common CI); return ONLY compact JSON {root_cause, workaround, is_known_error, permanent_fix}. Retry
   until the content is non-empty.
4. Publish: upsert the problem (root_cause, state known_error, known_error true, workaround) and a
   known_errors row (published true); stamp incidents.problem_id on the clustered incidents. COMMIT
   before reading any of those rows back — DataK3 has no read-your-writes inside an open transaction.
   If link_change: spawn a changes row in state "assess" (NOT "new" — change management rejects "new"
   with 409 forever), with the problem_id back-reference, then set problems.related_change_id and
   known_errors.permanent_fix_change_id.
5. No permission gates on any of it: clustering and publishing a known error are the service desk's
   ordinary work. The gate that matters is itsm:change:approve, downstream at the CAB.
6. Deploy an image-mode Ignite app `itsm-problem-clusterer` — an HTTP server (GET /healthz, POST /cluster)
   built from a Dockerfile (--dockerfile-path, --port 8080, --health-path /healthz), own service account
   with k3.editor + ignite.model-user + ignite.app-developer, DODIL_SERVICE_ACCOUNT_ID = the cli-
   serviceAccountId (not the uuid), knobs CLUSTER_MIN/CLUSTER_THRESHOLD/AUTO_KNOWN_ERROR/LINK_CHANGE as
   env. It writes over the Postgres wire (pg.uk-lon-1.dodil.io), every re-writable key with
   INSERT ... ON CONFLICT DO UPDATE. There is no server-side scheduler: pick either a pinned always-on
   app (--reserved 1 --max-replicas 1) with its own loop, or an external scheduler, and say which.

Get the code

The package is a real download — code/itsm-problem-management/v1.tar. This post is a walkthrough of exactly those files; the tarball is source only (no Dockerfile — the deploy story is Step 5 / Ship it):

models.py          # SQLAlchemy — problem_incidents + known_errors (owned) · incidents/problems/changes (itsm/core masters, stubbed)
                   #   + the patched Vector subclass every ITSM package that reads an embedding ships
routes.py          # FastAPI    — incident/problem CRUD + cluster→RCA-gate→publish (vector+Models) + similar (vector) + chain (JOIN)
db.py              # lazy engine + the ON CONFLICT upsert helper every route uses (shared, byte-identical)
sa_token.py        # mints + refreshes the service-account client_credentials token used as the pg-wire password
auth.py            # header-trust role gate — reads what the gateway injected. NO verifier, no JWKS (shared, byte-identical)
.env.example       # BUCKET + PG_HOST/PG_PORT + DODIL_TOKEN + the SA gate creds. No issuer, no audience — nothing to configure
requirements.txt   # sqlalchemy, psycopg[binary], pgvector, fastapi, uvicorn, pydantic, httpx   (no pyjwt — nothing to verify)
README.md          # run it, and the DataK3 rules baked into the code
EXTENDING.md       # the pattern for adding a workflow route
PLATFORM.md        # the platform invariants you COPY rather than generate — identical in every DODIL package

sa_token.py is lazy on purpose: no token is minted at import, so app.openapi() builds with no credentials at all and CI can generate the API client without secrets. And PLATFORM.md is the one to read before you copy any of this into a customer build — every line in it is a scar from a real failure on this platform, and it travels inside the tar so whoever downloads the code gets the rules with it.

Run it — point .env at your bucket, create the tables from the models, serve:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
 
cp .env.example .env      # then set DODIL_TOKEN (your `dodil auth login` token)
# BUCKET defaults to "itsm"; create it once (Step 0), then create the tables from the models:
python -c "import db, models; models.Base.metadata.create_all(db.engine)"
 
# there is no gateway in front of a laptop, so opt in to a stub identity (NEVER set this deployed):
export DEV_ALLOW_ANON=1
uvicorn routes:app --reload
# POST /incidents · POST /incidents/{id}/cluster · POST /incidents/similar
# GET  /problems/{id}/chain · GET /known-errors/{problem_id}

models.Base.metadata.create_all is the ORM tab's every class at once — the same tables Steps 0–1 built by CLI, created from the natural-key models with no migration tool. The root-cause gate (/cluster with auto_known_error) also needs a service account granted ignite.model-user (set DODIL_SERVICE_ACCOUNT_ID = the cli-… serviceAccountId, not the uuid); leave it blank and cluster with auto_known_error=false to run the vector + SQL path without Models.

Connect your tools

Everything this build wrote lives in one DataK3 bucket, reachable by your own stack — not just the CLI. data connect itsm prints the endpoints; point your tools straight at the same rows:

  • SQL over the Postgres wire — psql, psycopg/asyncpg (Python), node-postgres (TS), a BI tool.
  • Vector — pgvector (<=>) over the same wire, or a Qdrant/Pinecone client against the same incidents rows.

Full, live-validated walkthrough: Connect your tools.

Ship it

itsm-problem-clusterer is an image-mode Ignite app — an HTTP server you POST a seed incident to on /cluster. Give it a least-privilege service account (the three roles in Step 5, and set DODIL_SERVICE_ACCOUNT_ID to the cli-itsm-problem-clusterer-sa serviceAccountId, not the uuid), inject the knobs as env, and deploy its Dockerfile with dodil ignite app deploy itsm-problem-clusterer --code ./clusterer --dockerfile-path Dockerfile --port 8080 --health-path /healthz. The platform builds the image on deploy (Lane B) — no --runtime python, no separate build step. Because there is no server-side scheduler, drive the tick yourself or pin a warm poll loop (--reserved 1 --max-replicas 1) — pick one deliberately, per the note in Step 5. The full lifecycle — DODIL git → CI checks → a scanned image in the registry → versioning and rollback — is walked end to end in Ship a DODIL App.

The suite — seven components, one app

This package runs standalone (uvicorn routes:app) — that is what this post walks through, and the code/itsm-problem-management download is still exactly that.

Deployed, the seven ITSM components compose into one app. code/itsm-suite-app is a single FastAPI with a router per component, one canonical models.py covering all 23 tables, and plain imports — no importlib loader — over one bucket (itsm) and one dodil-appid pool, so seven components mean one sign-in and one bill. It ships by the ordinary git cycle (repo → CI → registry → CD): Ship a DODIL app. One app rather than seven is the ERP default; you split only for a stated reason — a public surface against a private engine, independent scaling, a distinct trust boundary — and ITSM has none of those.

That one canonical models.py is not a packaging convenience, and this post is the evidence. All three bugs above are disagreements between two copies of the same model: an attribute named differently in one package, a change lifecycle understood differently by producer and consumer. Unify the models and the disagreements have nowhere left to live. It is also how they were found — the ## Test you just read is the first time this component ever ran with the other six present.

Conclusion

The same outage stops billing MTTR three times. The recurrence is found by meaning (cosine KNN over the incidents you already embedded), the root cause is synthesised by a kimi-k2.6 gate and pinned to a known-error row an on-call can search, and the permanent fix is already a change in flight — all rows in the one bucket your ITSM lives in, one JOIN from each other. Tune the recurrence bar with cluster_min, the cluster radius with cluster_threshold, and hand the fix to change management with link_change — the whole engine is one Ignite app over one copy of your rows.

Next steps: