What you'll build: an incident-triage engine that takes a fresh incident and, in one pass, dedups it against similar open tickets (vector), pulls the blast radius of its affected CI from the CMDB (graph), asks a kimi-k2.6 gate for a category + priority, and routes it to the team that fixed the nearest precedent — writing the verdict to an incident_triage table and advancing the incident new → triaged. It's one Ignite app over the same DataK3 bucket your ITSM already lives in (see Build a ServiceNow-style ITSM on DataK3); it consumes core incidents/cis/services/groups + incidents.embedding + the cmdb_impact graph, and adds two tables of its own.

The problem — and why it matters

The on-call engineer's most expensive minutes are the first ten. A new incident lands in the queue as a line of free text — "Order database connections exhausted, checkout failing" — and a human has to decide, cold: is this a duplicate of the outage we're already fighting? how bad is it? who owns it? That triage is done by hand, per ticket, at 2am, and it's where MTTR is won or lost. An ITSM platform charges per fulfiller seat (300–2,000 agents at ~$100–$150/mo — seven figures a year) and still leaves that judgement to the person staring at the queue.

Three questions decide the triage, and each is a different query pillar over the same incident rows:

  • "Have we seen this before?" — a semantic search over past incidents (Vector). A near-duplicate of an open incident should be linked, not opened again; the nearest resolved incidents are the precedent — and the team that fixed them is the right assignee.
  • "What breaks if this CI is down?" — a blast-radius traversal of the CMDB (Graph). A symptom on a criticality-1 database that takes down three tiers is a P1, not a P3.
  • "What is it, and how urgent?" — a classification (Models). A low-cost model reads the symptom + the blast radius and returns a structured category/priority, so the queue self-sorts and humans spend their night on the P1s.

The money: a self-triaging queue deflects duplicates, routes on the first touch, and grades severity from real dependency data — instead of paying a senior engineer to do it by hand while the clock runs. And because the tickets, the CMDB graph, and the similar-incident vectors are one copy of the rows, the whole verdict is one connection, not a nightly sync between Postgres, Neo4j, and Pinecone.

PieceLands inPillar / runs on
The dedup + precedent searchreads core incidents.embeddingVector (jina-embeddings-v4, cosine)
The blast-radius signalreads the cmdb_impact graphGraph (graph_khop / Bolt)
The classificationkimi-k2.6 verdict → incident_triageIgnite Models (the gate)
The verdict + dedup linktable incident_triageSQL (one row per incident)
The audit trailtable incident_work_notesSQL (every state change)
The triage enginereads masters, writes incident_triage + incidents.stateIgnite app itsm-triage-engine

NOTE

Connect the DODIL MCP once, then every step shows an Ask your agent tab (the default — DODIL is agent-native) and a CLI tab. Install itsm/core first if you're building the full suite — it owns the masters (incidents/cis/services/groups) and the inline incidents.embedding this skill reads, and itsm/cmdb-blast-radius owns the cmdb_impact graph. Standalone, Step 0 stubs those masters + the graph, so you can run this end to end without the rest of the suite. One catch on a standalone stub: it creates the incidents.embedding column but nothing fills it (populating embeddings is itsm/core's job — it embeds each incident on upsert). Until you seed embedded history, the dedup/precedent search returns zero neighbors silently — it is not an error, just an empty vector index. Either run itsm/core's embed-and-seed step first, or seed a few resolved incidents with jina-embeddings-v4 embeddings here so precedent has something to match.

Prerequisites

  • The dodil CLI (dodil auth login) or the DODIL MCP connected to your agent.
  • export BUCKET=itsm — the same bucket your ITSM masters (incidents, cis, services, groups) already live in.
  • The live model ids (confirm with dodil ignite models list): chat kimi-k2.6, embeddings jina-embeddings-v4 (2048-dim).
  • The itsm/core masters + incidents.embedding, and the cmdb_impact graph from itsm/cmdb-blast-radius. If you don't have them yet, Step 0 stubs the minimum this skill reads.

Step 0 — Stub the masters you consume (skip if you have itsm/core)

This skill triages the ITSM masters; it doesn't own them. If you built itsm/core + itsm/cmdb-blast-radius (or the full suite), those tables and the cmdb_impact graph already exist — skip to Step 1. Standalone, create the four masters this skill reads — cis, incidents (with the inline embedding VECTOR(2048)), services, groups — with the same column definitions core uses (every non-key column nullable:true, so a partial row never trips NotNullViolation), then the CMDB edges + the reverse cmdb_impact graph the blast-radius signal needs.

You

Create the itsm bucket, then four merge-keyed master tables with all non-key columns nullable: cis (key id: name, ci_type, environment, owner_group, service_id, business_criticality(int), status); incidents (key id: number, short_description, description, ci_id(bigint), service_id, state, priority, impact, urgency, category, subcategory, assignment_group, assigned_to, problem_id(bigint), opened_at, resolved_at, resolution, embedding VECTOR(2048)); services (key service_id: name, business_service, owner_group, tier(int), sla_id); groups (key group_id: name, manager, email, on_call(boolean)).

DODIL MCP tools called
data_bucket_create→data_table_create
Agent

Created bucket itsm and 4 master tables — cis (pk id), incidents (pk id, incl. embedding VECTOR(2048)), services (pk service_id), groups (pk group_id) — all non-key columns nullable, so a partial seed row writes cleanly. These are the exact itsm/core masters; if you already ran itsm/core they're here and this step is a no-op.

Now seed the demo estate — the same one every ITSM component is validated against: 13 CIs (apps, databases, a host, a CDN edge) wired by 14 typed edges, 4 business services, 5 groups, and 13 incidents. Six are resolved history (each with its fix and the group that delivered it), the rest are open. Each incident's embedding is a VECTOR(2048) from jina-embeddings-v4; upsert one row per call (a batched frame of 2048-dim vectors exceeds the gateway's first-frame size).

You

In the itsm bucket, upsert the 13-CI demo estate: pg-orders-primary and pg-orders-replica and redis-session databases; checkout-api, orders-api, payments-gateway, web-storefront, accounts-api, reporting-etl, mobile-app-bff, search-index, cdn-edge apps; host-app-01. Then 5 groups (g-orders, g-platform, g-dba, g-network, g-cab) and 4 services (svc-orders Order Management, svc-checkout Online Checkout, svc-accounts Customer Accounts, svc-reporting Internal Reporting). Then for each incident embed short_description+description with jina-embeddings-v4 and upsert one row per call: INC1001-1006 resolved history with resolution + assignment_group, INC1007-1013 open. Include all columns — a full-row upsert that omits a declared column will not persist.

DODIL MCP tools called
data_table_upsert→ignite_models_embed
Agent

Upserted 13 CIs, 5 groups, 4 services and 13 incidents (INC1001-1006 resolved, INC1007-1013 open). Each incident carries a 2048-dim jina-embeddings-v4 vector in embedding. This is the estate every ITSM component is validated against — one bucket, shared by all seven.

WARNING

incidents.service_id is a String, not a BigInteger — and getting that wrong was one of the six integration bugs. The masters key services by a string (svc-orders), and every component mapped service_id as String — except itsm-problem-management, which mapped it BigInteger. Alone, that package was perfectly self-consistent: its own stub created the column as a bigint, its own tests wrote bigints into it, its own ## Test passed. The two readings only collide when a ticket written by one component is read by another on a shared bucket — which is exactly how the suite runs.

The lesson generalises past ITSM: a column's type is part of a component's contract with its neighbours, not a private choice, and a stubbed master is a lie you cannot detect until a real one turns up. That is why the deployed suite app carries one canonical models.py — seven routers, one set of table definitions — so the drift has nowhere to live. The rule it states, and the one to copy into your own build: the ORM attribute name IS the column name, and a mapped type is the column's type (see itsm/core).

Step 1 — The triage tables (the verdict store + the audit trail)

This skill owns two tables. incident_triage is the verdict store — one row per incident, merge-keyed on incident_id, so a re-triage upserts in place (never a duplicate). It carries the classification (category/priority/assignment_group), the evidence (similar_incident_ids, blast_radius_json), and the dedup link (is_duplicate/master_incident_id). incident_work_notes is the audit trail — every state change and work note, one row per note_id.

NOTE

data table create makes every non-PK column NOT NULL by default — set "nullable":true on any optional column. Here every non-key column is nullable: a dedup-only row leaves category/priority/assignment_group null (the incident was linked, not classified), so a write to a NOT-NULL column would 500 with NotNullViolation.

You

In the itsm bucket, create two merge-keyed tables with all non-key columns nullable. incident_triage (key incident_id): category, subcategory, priority, assignment_group, confidence(double), similar_incident_ids(json), blast_radius_json(json), rationale, model_id, is_duplicate(boolean), master_incident_id(bigint), triaged_at. incident_work_notes (key note_id): incident_id(bigint), kind, author, body, from_state, to_state, ts.

DODIL MCP tools called
data_table_create
Agent

Created incident_triage (key incident_id, 13 columns — verdict, evidence, and dedup link, all nullable) and incident_work_notes (key note_id, 8 columns). Upserts are idempotent, so re-triaging an incident updates its one row.

Step 2 — Similar-incident dedup + precedent retrieval (Vector)

The incidents.embedding column core populated is the semantic index — no Pinecone, no second copy. Two questions, same column:

Dedup — is this a repeat of an open ticket? KNN the nearest open incident (state new/triaged, not self). If it's inside dedup_threshold (0.20 cosine), the new ticket is a duplicate — link it, don't open it fresh. The estate holds a real pair: INC1010 and INC1011 are two people reporting the same checkout 502, minutes apart. INC1011's nearest open neighbour is INC1010 at cosine 0.0785 — well inside the threshold:

You

In the itsm bucket, find the nearest OPEN incident to INC1011 (id 1011, exclude itself) by cosine over incidents.embedding — pgvector KNN with 1011's own embedding as the query, restricted to open states and to rows that actually carry an embedding.

DODIL MCP tools called
data_pg
Agent

Nearest open incident to INC1011 is INC1010 at cosine distance 0.0785 — well inside the 0.20 dedup threshold. INC1011 is a duplicate of INC1010: link it (master_incident_id=1010), do not triage fresh, and do not call the model at all.

NOTE

Filter embedding IS NOT NULL in the KNN. Not every open ticket has been embedded yet — a row created through plain CRUD carries a null vector until the embedder runs. Without the guard the whole KNN fails with list_cosine_distance: left argument can not contain NULL values, and the triage route 500s on a ticket that has nothing to do with the null row.

That threshold is doing real work in both directions. Run the same query for INC1009 — the fresh P1 this post triages — and its nearest open neighbour is INC1011 at 0.2326, above 0.20. Related, clearly: both are checkout-adjacent outages. But not the same ticket, so INC1009 gets a full triage rather than a link. One number, one decision, no human in the loop:

You

In the itsm bucket, find the nearest OPEN incident to INC1009 (id 1009) by cosine over incidents.embedding, excluding itself and rows with no embedding — I want to confirm it is above the 0.20 dedup threshold and therefore gets a full triage.

DODIL MCP tools called
data_pg
Agent

Nearest open incident to INC1009 is INC1011 at cosine 0.2326 — ABOVE the 0.20 dedup threshold, so INC1009 is not a duplicate and proceeds to full triage (precedent + blast radius + the classification gate).

Precedent — how was this fixed before, and by whom? KNN the nearest resolved incidents; the team that resolved the closest one is the default assignee (that's what auto_assign does). For INC1009, the three previous pg-orders-primary connection-pool outages surface in order — and all three are owned by g-dba:

You

In the itsm bucket, show the 3 nearest RESOLVED incidents to INC1009 by cosine over incidents.embedding, with their assignment_group and resolution — pgvector KNN using 1009's stored embedding as the query.

DODIL MCP tools called
data_pg→data_vsearch
Agent

Nearest resolved to INC1009: INC1003 (g-dba, PgBouncer pooling, cosine 0.1094), INC1005 (g-dba, 0.1369), INC1004 (g-dba, 0.1454). neighbors[0]=INC1003 -> route to g-dba. All three precedents agree on the owning team, which is the strongest possible routing signal.

The stored verdict records exactly that list — similar_incident_ids = [1003, 1005, 1004] — so the routing decision is auditable months later: these three tickets, this close, all fixed by this team.

The <=> operator needs a vector literal on the right-hand side — the handler fetches the target incident's stored embedding, formats the […] string, and interpolates it. (A correlated sub-SELECT of another row's vector does not drive the KNN; pass the literal.)

Step 3 — Blast radius of the affected CI (Graph)

Severity isn't in the ticket text — it's in the dependency graph. itsm/cmdb-blast-radius owns the reverse cmdb_impact graph (the impact-bearing ci_edges flipped); graph_khop('cmdb_impact', ci_id, 5) returns every CI that transitively depends on the failing one, hop-ranked, and you hydrate names by joining cis in the same statement. INC1009's affected CI is pg-orders-primary (CI 1), and a symptom there reaches 10 other CIs across 3 hops — the whole checkout path, the order path, accounts, and the reporting ETL:

You

In the itsm bucket, give me the full blast radius of pg-orders-primary (CI 1) from the cmdb_impact graph — graph_khop up to 5 hops, hop-ranked, hydrated with CI names and owner groups.

DODIL MCP tools called
data_pg
Agent

Blast radius of pg-orders-primary: 10 impacted CIs, max hop 3. Hop 1 — checkout-api, orders-api, accounts-api, reporting-etl, pg-orders-replica. Hop 2 — payments-gateway, web-storefront, search-index. Hop 3 — mobile-app-bff, cdn-edge. Ten CIs off one database is a P1 signal, not a P3.

Rolled up to business services — the sentence an incident commander actually says out loud — that blast touches all four: Online Checkout (5 CIs), Order Management (3), Customer Accounts (1) and Internal Reporting (1). This post consumes that traversal as a raw signal; the typed version — where dropping part_of composition edges changes the answer, because mobile-app-bff is reachable only over one — is itsm/cmdb-blast-radius's subject, and worth reading before you trust any blast number in a priority decision.

The same traversal in Cypher over Bolt — return the node variable (the graph plane hands back node keys; join cis for properties):

You

Same blast radius in Cypher over Bolt: from pg-orders-primary (id 1), follow impacts up to 5 hops on the cmdb_impact graph and return the impacted nodes.

DODIL MCP tools called
data_bolt
Agent

Returns the 10 impacted node keys — 3, 4, 8, 9, 11 at hop 1; 5, 6, 12 at hop 2; 10, 13 at hop 3 — the same set the graph_khop traversal returns, over the Bolt wire.

This blast set is a triage signal, folded into blast_radius_json and the gate prompt. It's optional: if you didn't install itsm/cmdb-blast-radius, the engine degrades to no-blast triage (the gate still classifies on the symptom alone).

Step 4 — The classification gate (Ignite Models)

Here's the gate. The engine assembles the signal bundle — the symptom, the affected CI (with its business_criticality), and the blast radius — and asks kimi-k2.6 for only compact JSON: a category from your list, a priority, and an assignment_group. The system prompt is rendered from the categories param. The model judges; the policy decides.

NOTE

kimi-k2.6 is a reasoning model — guard against empty content at this gate. It can spend its budget on hidden reasoning and return an empty content. Defend it two ways: end the prompt with Return ONLY compact JSON, no reasoning/preamble, and retry until the content is non-empty (a single retry is not enough for kimi-k2.6). Inside the Step 6 Ignite handler you also set max_tokens: 4096 on the raw api.dodil.io/v1 call (the interactive CLI/MCP path can't) and read the reply from data.content.

You

Classify INC1009 with kimi-k2.6 and return ONLY compact JSON {category one of [database, network, performance, availability, security], priority one of [P1,P2,P3,P4], assignment_group short}: symptom 'Order database connections exhausted, checkout failing', affected CI pg-orders-primary (database, business_criticality 1), blast radius 10 CIs across 3 hops including checkout-api, orders-api, accounts-api, reporting-etl, web-storefront.

DODIL MCP tools called
ignite_models_chat
Agent

kimi-k2.6 returned category "database" — a database incident. The category is in the allowed list, so the verdict is accepted; the deterministic impact×urgency matrix, not the model, sets the priority.

Two policy knobs shape what happens to that verdict, and the first is the one that keeps this defensible:

  • priority_scheme (default impact_urgency) — priority is derived from a deterministic impact × urgency matrix (ServiceNow-classic, auditable); the gate fills only category + assignment_group. A criticality-1 CI whose failure reaches 10 other CIs across all four business services is high impact × high urgency → P1. The model classifies; the matrix decides. Keep that line clean and you can answer "why is this a P1?" with a rule instead of a prompt. Flip to direct and the gate sets priority too — faster, and much harder to defend in a post-incident review.
  • auto_assign (default true) — the final assignment_group is neighbors[0]'s group, the team that fixed the nearest precedent — g-dba, which owned all three of INC1009's precedents. Set it false and the gate's suggestion is left for a human to confirm.

Step 5 — Write the verdict + advance the incident + log the work note

Now land it. For INC1009: upsert one incident_triage row (category, priority, the precedent group, the blast radius, the similar incidents), UPDATE the incident new → triaged with the classification, and log the state change to incident_work_notes. For the duplicate INC1011: an incident_triage row with is_duplicate=true + master_incident_id=1010 and no classification — it was linked, not triaged.

You

In the itsm bucket, write the triage verdict for INC1009: upsert incident_triage (category database, priority P1, assignment_group g-dba, similar_incident_ids [1003,1005,1004], blast_radius_json with the 10 impacted CI names, is_duplicate false, model_id kimi-k2.6), UPDATE incidents 1009 to state=triaged/category=database/priority=P1/assignment_group=g-dba, and log a new->triaged work note. Then write the dedup verdict for INC1011: incident_triage is_duplicate=true, master_incident_id=1010, category/priority null, plus a work note.

DODIL MCP tools called
data_table_upsert→data_pg
Agent

Upserted incident_triage for 1009 (triaged: database/P1/g-dba, 10-CI blast radius, precedents [1003,1005,1004]) and 1011 (is_duplicate=true, master 1010), advanced incident 1009 new->triaged, and logged wn-1009-triage and wn-1011-dup. The verdict is a row; the queue self-sorted.

The triaged queue is now a JOIN — an on-call sees exactly what's real, how bad, who owns it, and what's a duplicate of what:

You

In the itsm bucket, show every triaged incident joined to its verdict: incident_id, state, category, priority, assignment_group, is_duplicate, master_incident_id — ordered by incident_id.

DODIL MCP tools called
data_sql
Agent

1009 -> triaged / database / P1 / g-dba (is_duplicate false); 1011 -> is_duplicate true, master_incident_id 1010 (still new, never triaged fresh). The dedup gate deflected the repeat and the precedent routed the real one.

TIP

INC1011 never reached the model. The dedup arm answered from the vector pillar and stopped — no kimi-k2.6 call, no tokens billed, no latency. On a real service desk, where duplicate reports of the same outage are a large share of the queue, that is the cost story in one row: the cheapest classifier is the one you never call. The Models spend lands only on tickets that are genuinely new.

And the rows this step wrote are immediately other components' inputs. Minutes later the problem clusterer pulled INC1009 into the 0.25-radius cluster that became PRB2001, alongside the same INC1003/1004/1005 precedents this triage cited — two components reaching the same conclusion from the same vectors, with no export between them (itsm/problem-management).

Routes

The steps above are the triage engine's inner loop, run one call at a time. The download (see Get the code) fronts the same bucket with a small FastAPI app, routes.py — the verdict-store CRUD plus the one op a flat ticket queue can't do: dedup → precedent → blast → gate → write in a single call. This is what you deploy. Every route follows the same DataK3 rules the package bakes in.

The connection and the one write helper live in db.py. A DataK3 bucket is a Postgres endpoint — db name = the bucket, user = the literal token, password = your DODIL token — so there's no data connect step in code, just fixed region constants. upsert() is the only writer every route uses:

# db.py — INSERT ... ON CONFLICT DO UPDATE (idempotent keyed write)
def upsert(session, model, rows, key):
    keys = [key] if isinstance(key, str) else list(key)
    table = model.__table__
    cols = {c for r in rows for c in r}
    rows = [{c: r.get(c) for c in cols} for r in rows]
    stmt = pg_insert(table).values(rows)
    update_cols = [c.name for c in table.columns if c.name not in keys]
    if update_cols:
        stmt = stmt.on_conflict_do_update(
            index_elements=keys,
            set_={c: getattr(stmt.excluded, c) for c in update_cols},
        )
    else:
        stmt = stmt.on_conflict_do_nothing(index_elements=keys)
    session.execute(stmt)

Why it matters: on DataK3 a bare re-INSERT of an already-committed primary key raises duplicate-key 23505 — a plain INSERT is not an upsert on re-write. upsert makes a re-triage (a retry, a replayed webhook, a shard replay) land one incident_triage row per incident and one wn-<id>-triage note — never a duplicate. That's the whole reason re-triaging is safe.

The money op — POST /incidents/{incident_id}/triage. It fuses all four pillars over one session: dedup (vector), precedent (vector), blast radius (graph), the classification (Models), then the write. The dedup KNN passes the incident's own embedding as the query and reads the nearest open ticket; under the threshold it links and stops — no fresh triage:

# routes.py — the triage op (dedup arm): nearest OPEN incident, pgvector cosine distance
inc = s.get(Incident, incident_id)
qvec = inc.embedding                                   # the target's own embedding is the KNN query
dup = s.execute(
    select(Incident.id, Incident.embedding.cosine_distance(qvec).label("d"))
    .where(Incident.id != incident_id, Incident.state.in_(OPEN_STATES))
    .order_by("d").limit(1)
).first()
if dup and float(dup.d) < threshold:
    return _write_dup(s, incident_id, dup.id, float(dup.d))   # link to master, no fresh triage

Live-verified on the shared itsm bucket: INC1011 deduped to open INC1010 at cosine 0.0785 (< the 0.20 threshold) → linked, is_duplicate=true, never triaged fresh. Past that gate, precedent is the same KNN restricted to state='resolved', and the blast radius is a graph traversal — graph_khop projects node + hop_distance, resolves only as a top-level SELECT with an integer-literal start node (so the FastAPI-validated ci_id is inlined, not bound), and degrades to [] if cmdb_impact is absent:

# routes.py — the blast-radius helper (GRAPH): reverse impact traversal, hop-ranked
def _blast_radius(s: Session, ci_id: int) -> list[str]:
    try:
        rows = s.execute(text(
            "SELECT c.name FROM graph_khop('cmdb_impact', "
            f"{int(ci_id)}, 5) k JOIN cis c ON c.id = k.node ORDER BY k.hop_distance"
        )).scalars().all()
        return list(rows)
    except Exception:
        s.rollback()                                   # graph absent -> no-blast triage
        return []

Then the gate (kimi-k2.6, max_tokens: 4096, retry-until-non-empty — the reply is wrapped in data), and the write: upsert the incident_triage verdict, read-merge-upsert the full incident row to advance it new → triaged, and upsert the wn-<id>-triage work note — all keyed, so the whole op is idempotent. Live, INC1009 came back database / P1 / g-dba (the group that owned all three of its precedents), and re-running /triage left count(*) = count(DISTINCT incident_id) = 2 in incident_triage, one work note per incident.

The building blocks are their own routes too. POST /incidents/similar is the raw dedup/precedent KNN (pass an embedding, restrict states), GET /cis/{ci_id}/blast is the graph signal alone, and GET /queue is the triaged queue as one JOIN:

# routes.py — the similar-incident KNN (VECTOR) and the triaged queue (SQL)
@app.post("/incidents/similar")
def similar_incidents(q: SimilarIn, s: Session = Depends(db)):
    stmt = select(Incident.id, Incident.state, Incident.assignment_group,
                  Incident.embedding.cosine_distance(q.embedding).label("d"))
    if q.states:
        stmt = stmt.where(Incident.state.in_(q.states))
    if q.exclude_id is not None:
        stmt = stmt.where(Incident.id != q.exclude_id)
    rows = s.execute(stmt.order_by("d").limit(q.top_k)).all()
    return {"matches": [{"incident_id": i, "state": st, "assignment_group": g, "distance": float(d)}
                        for i, st, g, d in rows]}

Adding a new business operation touches only routes.py (and maybe models.py) — the plumbing in db.py is fixed. The pattern is one Pydantic *In schema + one @app.<verb> function: write via upsert, vector via cosine_distance(…), graph via graph_khop(…) (see EXTENDING.md in the package).

Auth — config at the edge, a role gate in the app

End-user login on Ignite is configuration, not code. The ITSM deploys with the itsm-suite dodil-appid pool attached (user_pool: itsm-suite in .dodil/deploy.yaml), and the per-cluster Ignite gateway runs the entire browser login at the edge — PKCE S256 against the pool issuer, an AEAD-sealed host-only session cookie, single-flight refresh, EdDSA JWT verification against trust anchors this app does not hold — then injects the verified identity into every request it forwards:

  • X-Dodil-User — sub, email, connection, app_roles as plain JSON
  • X-Dodil-User-Jwt — the raw verified pool token, carrying the catalog-expanded permissions claim
  • X-Dodil-Auth-Source — pool (an app end-user) or platform (an operator or service-account invoke)

Any inbound copy of those headers is stripped first, on every mode and every principal, so a caller can never forge them.

What survives in the package is a small auth.py that ships no verifier — no JWKS client, no issuer/audience env, no crypto dependency, and no pyjwt in requirements.txt. It reads the injected header and keeps the one job the app still owns: role-based gating.

# auth.py — a header-trust reader, not a verifier (the gateway already checked the JWT)
def current_user(request: Request) -> dict:
    raw = request.headers.get("x-dodil-user")   # {"sub","email","connection","app_roles"}
    ...                                          # + permissions read off x-dodil-user-jwt
    raise HTTPException(401, "end-user login required — no X-Dodil-User from the gateway")
 
def require_permission(perm: str):
    """Gate a route on a pool permission: Depends(require_permission("itsm:change:approve"))."""
    def _dep(user: dict = Depends(current_user)) -> dict:
        if perm not in user["permissions"] and perm not in user["roles"]:
            raise HTTPException(403, f"missing permission: {perm}")
        return user
    return _dep

The gate audit — four permissions across seven components

ITSM permissions are namespaced <module>:<object>:<verb>, so one customer pool can carry every ERP module's roles without collision (itsm:change:approve is not crm:change:approve). Auditing all seven components left exactly four gates standing:

PermissionGatesWhy this one and not the rest
itsm:change:approvechange-management: POST /changes/{id}/assess, POST /changes/{id}/transition, POST /change-policiesaccepting the risk of a production change. change_approvals.decided_by records the gateway-vouched user
itsm:major:declaremajor-incident: POST /major/declaredeclaring a major incident pages the org. bridges.declared_by records who
itsm:incident:resolvemajor-incident: POST /major/bridges/{id}/statethe only route in the module that writes incidents.state='resolved' — what stops the SLA clock
itsm:cmdb:rebuildcmdb-blast-radius: POST /graph/assembleit TRUNCATEs impacts + service_map and DROP+CREATEs both graphs; every other component's blast radius is computed from what it leaves behind

Four of the seven components ended with zero gates, on purpose — itsm-core, itsm-incident-management, itsm-problem-management and itsm-sla-management. CMDB CRUD, triage and clustering are the service desk's ordinary work. A permission that every agent on the desk must hold does not restrict anything; it just adds a row to the role catalog and a way for the queue to stop moving at 3am because someone's pool role was mis-set. Ceremony is exactly what an auditor discounts.

So this component's earlier design was wrong, and the audit deleted it. There is no incidents:triage permission any more. Triaging a ticket is what a service-desk agent is for; gating it on a permission every agent holds bought nothing and cost an outage mode. Two sibling gates went the same way in the same pass — problems:write (clustering is ordinary analysis) and cmdb:write (it sat on an idempotent per-CI precompute, which is safe to re-run by construction). Running the other way, the audit added one the roadmap had not predicted: itsm:cmdb:rebuild, because the route it guards is the most destructive operation in the module.

The test that survives is worth stealing: gate the routes that accept risk or that are hard to undo, and nothing else. Approving a change accepts production risk. Declaring a major incident pages the company. Resolving a ticket stops a contractual clock. Rebuilding the CMDB graph destroys and recreates it. Triage does none of those — it writes an idempotent verdict row you can recompute at will.

Concretely, in this package every route takes Depends(current_user) and nothing more — the identity is still there, attributed and logged, but no permission is checked:

# routes.py — signed in, not gated. current_user attributes the work; nothing here accepts risk.
from auth import current_user
 
@router.post("/incidents/{incident_id}/triage")
def triage(incident_id: int, q: TriageIn, user=Depends(current_user),
           s: Session = Depends(db)):
    ...
 
@router.get("/queue")
def triaged_queue(user=Depends(current_user), s: Session = Depends(db)):
    ...   # any signed-in fulfiller sees the queue

The pool, created once

Create the pool once for the whole suite, with the role catalog those four gates check — the service desk's real org chart expressed as permissions:

You

Create a dodil-appid pool itsm-suite with email+password, and set its role catalog: an agent does the ordinary service-desk work with no elevated permissions; a change-manager may approve changes; an incident-commander may declare a major incident and resolve one; a cmdb-admin may rebuild the CMDB graph.

Agent

Pool itsm-suite created — issuer https://appid.dodil.io/ihdiash/itsm-suite, audience pool:itsm-suite, email+password (local) enabled. Catalog set: agent holds NO permissions (it does the ungated work — triage, CMDB CRUD, clustering, the SLA views); change-manager = itsm:change:approve; incident-commander = itsm:major:declare + itsm:incident:resolve; cmdb-admin = itsm:cmdb:rebuild. A user's next token carries app_roles plus the catalog-expanded permissions claim the gates read.

Note that agent holds no permissions at all. That is not an oversight — it is the shape of the module. The service desk does the ungated work, which is most of the system, and the four permissions are held by the few people who accept risk on the organization's behalf.

The two-plane rule is unchanged: the pool identifies the user; the app still reaches DataK3 through its own service account (sa_token.py mints and refreshes a client_credentials token for the pg-wire password) — an app-user is never a bucket principal. Locally, with no gateway in front of uvicorn routes:app, opt in to a stub identity with DEV_ALLOW_ANON=1; the stub carries no permissions unless you grant them (DEV_USER_PERMISSIONS=itsm:change:approve,…), so the gated routes elsewhere in the suite stay gated on a laptop too. Today it is email+password (local); oauth/oidc/saml corporate SSO switch on per pool later, no app change. Full flow: App authentication; the catalog mechanics: App roles.

The route with no identity at all — and why it generalises

One route in the suite takes neither a permission nor Depends(current_user), and the reasoning behind it is the most portable thing in this section:

POST /sla/tick carries no identity dependency at all. It is a machine heartbeat, called by a service account over a platform invoke, which carries no X-Dodil-User. A current_user dependency there would 401 the clock; the engine would stop and every incident's breach flags would silently go stale — the failure mode of an SLA system that reports green while it is blind. Exposure is the ingress's job (public_invoke=false), not a user permission. This applies to every recompute route in every module — a rollup, a re-materialization, an embedding backfill: if a machine calls it, gate the door, not the caller.

The same logic governs itsm-triage-engine below. It is invoked by your incident intake, not by a browser, so it authenticates as a service account and is kept private at the ingress — not behind a user permission that no machine will ever hold.

Get the code

The package is a real download — code/itsm-incident-management/v1.tar. This post is a walkthrough of exactly those files; the tarball is source only (no Dockerfile — deploy is in Step 6 / Ship it):

models.py          # SQLAlchemy — incident_triage + incident_work_notes (owned) + the itsm/core masters it reads
routes.py          # FastAPI    — CRUD + the triage op (dedup vector + precedent + blast graph + kimi-k2.6 gate)
db.py              # the lazy engine + the ON CONFLICT upsert helper every route uses
sa_token.py        # mints/refreshes the service-account client_credentials token (the pg-wire password)
auth.py            # header-trust role gate — reads what the gateway injected; NO verifier, no JWKS
.env.example       # BUCKET + PG_HOST/PG_PORT + DODIL_TOKEN + the SA gate creds + the triage policy knobs
requirements.txt   # sqlalchemy, psycopg[binary], pgvector, fastapi, uvicorn, pydantic, httpx
README.md          # what it is, how to run it
EXTENDING.md       # the pattern for adding a workflow route
PLATFORM.md        # the platform invariants — every line a scar from a real failure on this platform

Two of those are worth a sentence. sa_token.py is lazy on purpose: no token is minted at import, so app.openapi() builds with no credentials at all and CI can generate the API client without secrets. PLATFORM.md ships the platform rules inside the tarball — merge-key writes, ON CONFLICT instead of bare INSERT, commit-before-read, COPY *.py, the numeric non-root USER — so whoever downloads the package gets the rules along with the code instead of having to rediscover them. And note what is not in .env.example any more: no APPID_ISSUER, no APPID_AUDIENCE, no JWKS URL. The gateway does the login; there is nothing to configure.

Run it — point .env at your bucket, create the tables from the models, serve:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
 
cp .env.example .env      # then set DODIL_TOKEN (your `dodil auth login` token)
# BUCKET defaults to "itsm"; create it once (Step 0), then create the tables from the models:
python -c "import db, models; models.Base.metadata.create_all(db.engine)"
 
uvicorn routes:app --reload
# GET /incidents/{id} · GET /triage/{id} · POST /incidents/{id}/triage · POST /incidents/similar
# GET /cis/{ci_id}/blast · GET /queue

models.Base.metadata.create_all is the ORM tab's every class at once — the same tables Steps 0–1 built by CLI, created from the natural-key models with no migration tool. The /triage route calls Models, so it also needs the service-account creds in .env (DODIL_SERVICE_ACCOUNT_ID = the cli-… serviceAccountId, not the uuid); the read-only routes need only DODIL_TOKEN.

Step 6 — The triage engine on Ignite (itsm-triage-engine)

The calls above are the engine's inner loop. In production one Ignite app, itsm-triage-engine, does it per incident: a POST /triage with {incident_id} dedups, retrieves precedent, folds in the blast radius, calls the gate, and writes the verdict — all over one Postgres-wire connection (SQL + pgvector <=> + graph_khop() are all just SQL there). It's a separate workload, so it gets its own service account — and because it both writes DataK3 and calls Models, it needs three live roles (confirm the exact names with dodil auth service-account list-roles):

  • k3.editor — write incident_triage/incident_work_notes and update incidents.
  • ignite.model-user — call kimi-k2.6 from inside the handler (token-billed).
  • ignite.app-developer — the deploy/invoke identity for the app itself.

NOTE

There is no ignite.developer role (an older doc named one) — the live catalog splits it into ignite.app-developer (deploy/invoke) and ignite.model-user (call models). A pure-SQL handler (like the SLA monitor) needs only k3.editor + ignite.app-developer; this one calls Models, so it needs all three. Confirmed live 2026-09-02 (org IHDIASH).

This ships as an image-mode Ignite app: a plain HTTP server (GET /healthz for the probe, POST /triage for the work), packaged by a Dockerfile and built on deploy — not a handler(payload, ctx) compile-mode function. Three things matter and each is a line below:

  • The Models call is the real OpenAI-compatible endpoint (api.dodil.io/v1), authed with a service-account token, max_tokens: 4096 (kimi-k2.6 is a reasoning model — a low budget returns empty content), and the reply read from data.content; empty content is retried until non-empty.
  • Reads/writes go over the drop-in Postgres wire (pg.uk-lon-1.dodil.io:5432, dbname=<bucket>, user=token, password=<the SA access token>) via psycopg — there is no K3 HTTP API. Re-triaging an incident re-writes the same incident_id / note PK, so the writes are INSERT … ON CONFLICT (<pk>) DO UPDATE (a bare re-INSERT of an already-committed PK raises duplicate-key 23505 — a plain INSERT is not an upsert on re-write; DuckDB pg-wire supports ON CONFLICT). Writes retry on SerializationFailure.
  • Every call to id.dodil.io / api.dodil.io sets an explicit User-Agent — stdlib urllib's default is Cloudflare-banned (HTTP 403 "error code: 1010").
# server.py — itsm-triage-engine, an IMAGE-mode Ignite app (HTTP server on $PORT).
#   GET  /healthz -> {"status":"ready"}      (probe; no auth)
#   POST /triage  -> {"incident_id": N} -> dedup + precedent + blast + gate -> writes the verdict
 
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
from psycopg import errors as pg_errors
 
# --- hoisted knobs: mirror the skill params, injected as env at deploy ---
CATEGORIES      = os.environ.get("CATEGORIES", "database,network,performance,availability,security").split(",")
PRIORITY_SCHEME = os.environ.get("PRIORITY_SCHEME", "impact_urgency")   # impact_urgency | direct
AUTO_ASSIGN     = os.environ.get("AUTO_ASSIGN", "true").lower() == "true"
DEDUP_THRESHOLD = float(os.environ.get("DEDUP_THRESHOLD", "0.20"))       # cosine; below -> duplicate
MODEL_ID        = os.environ.get("MODEL_ID", "kimi-k2.6")
BUCKET          = os.environ["BUCKET"]
SA_ID           = os.environ["DODIL_SERVICE_ACCOUNT_ID"]   # the cli-… serviceAccountId, NOT the uuid
SA_SECRET       = os.environ["DODIL_SERVICE_ACCOUNT_SECRET"]
PG_HOST         = os.environ.get("PG_HOST", "pg.uk-lon-1.dodil.io")
PG_PORT         = int(os.environ.get("PG_PORT", "5432"))
 
ID_URL     = "https://id.dodil.io/realms/dodil/protocol/openid-connect/token"
MODELS_URL = "https://api.dodil.io/v1/chat/completions"
UA         = "itsm-triage-engine/1.0"   # explicit UA — stdlib urllib's default is Cloudflare-banned (403 1010)
OPEN_STATES = ("new", "triaged", "in_progress")
 
SYS = ("You are an ITSM triage assistant. Given the incident, its affected CI, and its blast radius, "
       "reply with ONLY compact JSON: {\"category\": one of [" + ", ".join(CATEGORIES) + "], "
       "\"priority\": one of [P1, P2, P3, P4], \"assignment_group\": short}. "
       "No prose, no reasoning, no preamble. Return ONLY compact JSON, no reasoning/preamble.")
 
def _now(): return datetime.now(timezone.utc).isoformat()
 
def _http_post(url, data, headers, form=False):
    headers = {"User-Agent": UA, **headers}          # the UA is required (see above)
    if form:
        body = urllib.parse.urlencode(data).encode()
        headers["Content-Type"] = "application/x-www-form-urlencoded"
    else:
        body = json.dumps(data).encode()
        headers["Content-Type"] = "application/json"
    req = urllib.request.Request(url, data=body, headers=headers, method="POST")
    with urllib.request.urlopen(req, timeout=90) as r:
        return json.loads(r.read().decode())
 
def _token():                                        # OIDC client_credentials -> access token
    out = _http_post(ID_URL, {"grant_type": "client_credentials",
                              "client_id": SA_ID, "client_secret": SA_SECRET},
                     headers={}, form=True)
    return out["access_token"]
 
def _gate(token, bundle):                            # kimi-k2.6: max_tokens 4096, retry until non-empty
    for attempt in range(5):
        out = _http_post(MODELS_URL,
                         {"model": MODEL_ID, "max_tokens": 4096,
                          "messages": [{"role": "system", "content": SYS},
                                       {"role": "user", "content": bundle}]},
                         headers={"Authorization": f"Bearer {token}"})
        env = out.get("data", out)                   # reply is wrapped in "data" on this platform
        content = (env.get("content")                # .data.content (CLI/MCP shape) …
                   or env.get("choices", [{}])[0].get("message", {}).get("content", ""))  # …or OpenAI shape
        if content and content.strip():
            return _extract_json(content)
        time.sleep(0.4 * (attempt + 1))              # empty content -> retry (once is not enough)
    raise ValueError("gate returned empty content after retries")
 
def _extract_json(text):
    text = text.strip()
    if text.startswith("```"):
        text = re.sub(r"^```[a-zA-Z]*\n?", "", text); text = re.sub(r"\n?```$", "", text).strip()
    m = re.search(r"\{.*\}", text, re.DOTALL)
    return json.loads(m.group(0) if m else text)
 
def _pg(token):                                      # drop-in Postgres wire: db=bucket, user=token, pw=SA token
    return psycopg.connect(host=PG_HOST, port=PG_PORT, dbname=BUCKET,
                           user="token", password=token, sslmode="require",
                           connect_timeout=20, autocommit=False)
 
def _vlit(vec):                                      # a pgvector literal '[f1,f2,…]' the <=> operator needs
    return "[" + ",".join(repr(round(float(x), 6)) for x in vec) + "]"
 
def _priority(criticality, blast_size, model_priority):
    if PRIORITY_SCHEME == "direct":                  # the model set priority
        return model_priority
    # impact_urgency: deterministic matrix — high impact (criticality-1 CI OR wide blast) x high urgency
    impact = "high" if (criticality is not None and criticality <= 1) or blast_size >= 2 else \
             "medium" if blast_size >= 1 else "low"
    return {"high": "P1", "medium": "P2", "low": "P3"}.get(impact, "P4")
 
def _retry(fn):
    for attempt in range(4):
        try:
            return fn()
        except (pg_errors.SerializationFailure, pg_errors.DeadlockDetected):
            if attempt == 3: raise
            time.sleep(0.4 * (attempt + 1))
 
def triage(incident_id):
    token = _token()
    with _pg(token) as conn, conn.cursor() as cur:
        cur.execute("""SELECT i.description, i.ci_id, c.name, c.business_criticality, i.embedding
                         FROM incidents i LEFT JOIN cis c ON c.id = i.ci_id WHERE i.id = %s""",
                    (incident_id,))
        row = cur.fetchone()
        if not row: return {"incident_id": incident_id, "error": "not_found"}
        desc, ci_id, ci_name, criticality, emb = row
        qvec = _vlit(emb)                            # the target's own embedding as the KNN query literal
 
        # 1. DEDUP — nearest OPEN incident (not self); below threshold -> link, don't triage fresh
        cur.execute(f"""SELECT id, embedding <=> '{qvec}' AS d FROM incidents
                         WHERE id <> %s AND state = ANY(%s) ORDER BY d LIMIT 1""",
                    (incident_id, list(OPEN_STATES)))
        dup = cur.fetchone()
        if dup and dup[1] < DEDUP_THRESHOLD:
            _write_dup(token, incident_id, dup[0], dup[1]); return {
                "incident_id": incident_id, "is_duplicate": True, "master_incident_id": dup[0],
                "cosine": round(dup[1], 4)}
 
        # 2. PRECEDENT — nearest RESOLVED incidents (fix + owning team)
        cur.execute(f"""SELECT id, assignment_group FROM incidents
                         WHERE state = 'resolved' ORDER BY embedding <=> '{qvec}' LIMIT 3""")
        neighbors = cur.fetchall()
 
        # 3. BLAST — reverse impact traversal of the affected CI (optional: skips if no cmdb_impact graph)
        blast = []
        try:
            cur.execute("""SELECT c.name FROM graph_khop('cmdb_impact', %s, 5) k
                             JOIN cis c ON c.id = k.node ORDER BY k.hop_distance""", (ci_id,))
            blast = [r[0] for r in cur.fetchall()]
        except Exception:
            conn.rollback()                          # graph absent -> no-blast triage
 
    # 4. GATE — classify (outside the txn; the model call is slow)
    bundle = (f'Incident: "{desc}". Affected CI: {ci_name} (business_criticality {criticality}). '
              f'Blast radius (impacted CIs): {", ".join(blast) or "none"}. '
              f'Return ONLY compact JSON, no reasoning/preamble.')
    verdict = _gate(token, bundle)
    assignment_group = (neighbors[0][1] if (AUTO_ASSIGN and neighbors) else verdict.get("assignment_group"))
    priority = _priority(criticality, len(blast), verdict.get("priority"))
 
    # 5. WRITE — verdict + advance the incident + work note (idempotent via ON CONFLICT on stable PKs)
    _write_triage(token, incident_id, verdict["category"], priority, assignment_group,
                  [n[0] for n in neighbors], blast)
    return {"incident_id": incident_id, "category": verdict["category"], "priority": priority,
            "assignment_group": assignment_group, "blast_radius": blast,
            "similar_incidents": [n[0] for n in neighbors], "is_duplicate": False}
 
def _write_triage(token, iid, category, priority, group, similar, blast):
    def _w():
        with _pg(token) as conn, conn.cursor() as cur:
            # incident_triage is keyed on incident_id, incident_work_notes on note_id (wn-<iid>-triage) —
            # both stable per incident, so a re-triage re-writes them: ON CONFLICT DO UPDATE, or a bare
            # re-INSERT of the committed PK raises duplicate-key 23505.
            cur.execute("""INSERT INTO incident_triage (incident_id, category, priority, assignment_group,
                             confidence, similar_incident_ids, blast_radius_json, model_id, is_duplicate, triaged_at)
                           VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
                           ON CONFLICT (incident_id) DO UPDATE SET category=EXCLUDED.category,
                             priority=EXCLUDED.priority, assignment_group=EXCLUDED.assignment_group,
                             confidence=EXCLUDED.confidence, similar_incident_ids=EXCLUDED.similar_incident_ids,
                             blast_radius_json=EXCLUDED.blast_radius_json, model_id=EXCLUDED.model_id,
                             is_duplicate=EXCLUDED.is_duplicate, triaged_at=EXCLUDED.triaged_at""",
                        (iid, category, priority, group, 0.9, json.dumps(similar),
                         json.dumps(blast), MODEL_ID, False, _now()))
            cur.execute("""UPDATE incidents SET state='triaged', category=%s, priority=%s,
                             assignment_group=%s WHERE id=%s""", (category, priority, group, iid))
            cur.execute("""INSERT INTO incident_work_notes (note_id, incident_id, kind, author, body,
                             from_state, to_state, ts) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
                           ON CONFLICT (note_id) DO UPDATE SET incident_id=EXCLUDED.incident_id,
                             kind=EXCLUDED.kind, author=EXCLUDED.author, body=EXCLUDED.body,
                             from_state=EXCLUDED.from_state, to_state=EXCLUDED.to_state, ts=EXCLUDED.ts""",
                        (f"wn-{iid}-triage", iid, "state_change", "itsm-triage-engine",
                         f"Auto-triaged: {category}/{priority} -> {group}. Blast: {', '.join(blast) or 'none'}.",
                         "new", "triaged", _now()))
            conn.commit()
    _retry(_w)
 
def _write_dup(token, iid, master, cosine):
    def _w():
        with _pg(token) as conn, conn.cursor() as cur:
            # same stable PKs (incident_id / note_id wn-<iid>-dup) — re-flagging a duplicate re-writes them.
            cur.execute("""INSERT INTO incident_triage (incident_id, confidence, similar_incident_ids,
                             model_id, is_duplicate, master_incident_id, triaged_at)
                           VALUES (%s,%s,%s,%s,%s,%s,%s)
                           ON CONFLICT (incident_id) DO UPDATE SET confidence=EXCLUDED.confidence,
                             similar_incident_ids=EXCLUDED.similar_incident_ids, model_id=EXCLUDED.model_id,
                             is_duplicate=EXCLUDED.is_duplicate, master_incident_id=EXCLUDED.master_incident_id,
                             triaged_at=EXCLUDED.triaged_at""",
                        (iid, 0.95, json.dumps([master]), MODEL_ID, True, master, _now()))
            cur.execute("""INSERT INTO incident_work_notes (note_id, incident_id, kind, author, body,
                             from_state, to_state, ts) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
                           ON CONFLICT (note_id) DO UPDATE SET incident_id=EXCLUDED.incident_id,
                             kind=EXCLUDED.kind, author=EXCLUDED.author, body=EXCLUDED.body,
                             from_state=EXCLUDED.from_state, to_state=EXCLUDED.to_state, ts=EXCLUDED.ts""",
                        (f"wn-{iid}-dup", iid, "work_note", "itsm-triage-engine",
                         f"Duplicate of INC{master} (cosine {cosine:.3f}); linked, no fresh triage.",
                         "new", "new", _now()))
            conn.commit()
    _retry(_w)
 
class Handler(BaseHTTPRequestHandler):
    def _send(self, code, body):
        payload = json.dumps(body).encode()
        self.send_response(code); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload))); self.end_headers()
        self.wfile.write(payload)
    def do_GET(self):
        if self.path == "/healthz": return self._send(200, {"status": "ready"})
        return self._send(404, {"error": "no_route", "path": self.path})
    def do_POST(self):
        if self.path != "/triage": return self._send(404, {"error": "no_route", "path": self.path})
        try:
            n = int(self.headers.get("Content-Length") or 0)
            body = json.loads(self.rfile.read(n) or b"{}")
            if "incident_id" not in body: return self._send(400, {"error": "missing incident_id"})
            return self._send(200, triage(int(body["incident_id"])))
        except urllib.error.HTTPError as e:
            return self._send(502, {"error": "upstream", "code": e.code,
                                    "body": e.read().decode(errors="replace")[:600]})
        except Exception as e:
            return self._send(500, {"error": type(e).__name__, "detail": str(e)[:600]})
    def log_message(self, *a): pass
 
if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8080"))
    print(f"itsm-triage-engine on 0.0.0.0:{port} scheme={PRIORITY_SCHEME} bucket={BUCKET}", flush=True)
    ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()

Only psycopg is a third-party dep; everything else is stdlib. The two sibling files that make it an image — ./triage-engine/Dockerfile and ./triage-engine/requirements.txt:

# ./triage-engine/Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY server.py .
ENV PORT=8080
EXPOSE 8080
CMD ["python", "server.py"]
# ./triage-engine/requirements.txt
psycopg[binary]==3.2.3

Give it a least-privilege identity, then deploy:

You

Create a service account itsm-triage-engine-sa, grant it k3.editor plus ignite.model-user and ignite.app-developer, then deploy my ./triage-engine app (image mode — its Dockerfile builds on deploy) to Ignite as itsm-triage-engine on port 8080 with health path /healthz, passing the service-account creds and the policy constants (CATEGORIES, PRIORITY_SCHEME impact_urgency, AUTO_ASSIGN true, DEDUP_THRESHOLD 0.20) as runtime env. Use the cli- serviceAccountId as DODIL_SERVICE_ACCOUNT_ID, not the uuid. Then POST {incident_id:1009} to /triage to smoke-test.

DODIL MCP tools called
auth_service-account_create→auth_service-account_grant-role→ignite_app_deploy
Agent

Created itsm-triage-engine-sa (serviceAccountId cli-itsm-triage-engine-sa), granted k3.editor + ignite.model-user + ignite.app-developer, built + deployed itsm-triage-engine (image:build, public FQDN on :8080, scale-to-zero). POST {incident_id:1009} to /triage returned category=database, priority=P1, assignment_group=g-dba, 10 impacted CIs in the blast radius.

NOTE

Deploy: image mode (Lane B). The deploy + /triage smoke-test above use image mode — a Dockerfile

  • --dockerfile-path, Kaniko build-on-deploy — not --runtime python (compile mode). This is the pattern validated live 2026-09-02 on the sibling CRM engine (crm-lead-scorer: deploys, serves /healthz + its route unauthenticated, writes durably). This build proved the engine's data/vector/graph/ Models logic live, one call at a time (Steps 2–5 and ## Test); the deploy wrapper is shown as code.

If dodil ignite app deploy returns IAM resource registration failed: broken pipe, deploy under a fresh app name — the half-created app can't be updated or deleted.

How the pillars map

One bucket, and the triage engine reaches across it — no second system to sync.

JobThe usual stackOn DataK3
The tickets + the verdictPostgres (ticket DB) + a CRM-style app objectincidents + incident_triage — merge-keyed SQL rows, re-triaged in place
"Have we seen this before?"Pinecone / Elasticsearch + an embedding pipelinecore incidents.embedding — a VECTOR(2048) column, data vsearch / pgvector <=>
"What breaks if this CI fails?"Neo4j (a separate CMDB)the cmdb_impact graph — graph_khop('cmdb_impact', …), Cypher over Bolt
The classificationA prompt bolted onto an external LLMkimi-k2.6 on Ignite Models — one auth context, token-billed
The engineA workflow runtime + connectorsIgnite itsm-triage-engine — scale-to-zero, its own service account

Because the tickets, the CMDB graph, and the similar-incident vectors are all in one bucket, the whole triage verdict is one connection — dedup KNN, blast traversal, and the write-back over the same psycopg cursor — not a nightly correlation between three stores.

Customize — the decisions this skill asks you

Q1 · categories — your incident taxonomy

"What incident categories should the gate choose from?" → The list the gate must pick from, rendered into the system prompt and the ## Test assertion. Default: database, network, performance, availability, security. Add or rename to match your catalog — the model's category is validated against exactly this set.

Q2 · priority_scheme — deterministic matrix or model?

"How is priority set — a deterministic impact × urgency matrix, or the model?"

  • impact_urgency (default) → priority is derived from an impact × urgency matrix (ServiceNow-classic, auditable); the gate fills only category + assignment_group. A criticality-1 CI whose failure reaches 10 other CIs across all four business services is high impact × high urgency → P1. Use it when a priority decision must be defensible.
  • direct → the model sets priority too (faster, less defensible). Sets PRIORITY_SCHEME=direct; the gate's priority is used verbatim.

Q3 · auto_assign — route to the precedent, or leave it?

"Auto-route to the team that fixed the nearest precedent, or leave it for a human?"

  • true (default) → assignment_group = neighbors[0]'s group (the team that resolved the closest resolved incident — g-dba for INC1009's connection-pool symptom). The queue self-routes.
  • false → the gate still classifies category/priority, but assignment_group is left for a human to confirm.

Q4 · dedup_threshold — how aggressive is dedup?

"Below what cosine distance to the nearest OPEN incident is a new ticket a duplicate?" → Default 0.20 (cosine). Below it, the incident is flagged is_duplicate + linked to master_incident_id instead of triaged fresh. The live estate shows the knob cutting both ways within a few hundredths: INC1011 → INC1010 at 0.0785 is well inside and gets linked, while INC1009's nearest open neighbour at 0.2326 is outside and gets a full triage. Lower = fewer false links (more genuine dups slip through as fresh tickets); higher = more aggressive deflection, and eventually two real outages collapsed into one ticket. This is the knob between noise and missed duplicates, and it is worth tuning against your own history rather than inheriting the default.

Industry variants (finserv / manufacturing) compose this skill with gate/policy tweaks. See the per-industry ITSM pages.

Test

Every command below ran live against DataK3 on 2026-09-08 (org IHDIASH) on the shared itsm bucket — the 13-CI / 14-edge / 4-service / 5-group estate, with all seven ITSM components installed together rather than on a private bucket of this component's own. That is the point: it is the composition that finds the bugs. This run is where the incidents.service_id type drift surfaced, and it is one of six integration bugs the joint validation turned up.

Both branches were exercised: the anchor {priority_scheme: impact_urgency, auto_assign: true} and the model-priority branch {priority_scheme: direct}. The dedup KNN linked INC1011 → INC1010 at cosine 0.0785, precedent put INC1003 at neighbors[0] (0.1094) with all three neighbours owned by g-dba, the kimi-k2.6 gate returned category: database, the impact_urgency matrix set P1, INC1009 advanced new → triaged, and a re-triage left count(*) = count(DISTINCT incident_id) = 2 in incident_triage — one work note per incident, idempotent. The bucket is still up.

export BUCKET=itsm
 
# 1. the gate returns a category from the allowed list; the MATRIX (not the model) sets priority
#    kimi-k2.6 -> category "database"; impact_urgency -> P1   (Step 4)
 
# 2. INC1009 -> triaged / database / P1 / g-dba (the group that owned all three precedents)
dodil data sql -b "$BUCKET" \
  "SELECT i.state, t.category, t.priority, t.assignment_group
     FROM incident_triage t JOIN incidents i ON i.id=t.incident_id WHERE t.incident_id=1009"
#  triaged | database | P1 | g-dba
 
# 3. the precedent trio — all three nearest RESOLVED incidents agree on the owning team
dodil data pg -b "$BUCKET" "
  SELECT n.id, n.assignment_group,
         ROUND(CAST(n.embedding <=> (SELECT embedding FROM incidents WHERE id=1009) AS DECIMAL(10,4)),4) AS d
    FROM incidents n WHERE n.state='resolved' ORDER BY d LIMIT 3"
#  1003 | g-dba | 0.1094 · 1005 | g-dba | 0.1369 · 1004 | g-dba | 0.1454
 
# 4. blast radius folded in — 10 impacted CIs off pg-orders-primary, max hop 3
dodil data pg -b "$BUCKET" \
  "SELECT count(*) AS impacted, max(k.hop_distance) AS max_hop
     FROM graph_khop('cmdb_impact', 1, 5) k JOIN cis c ON c.id=k.node"
#  10 | 3
 
# 5. INC1011 -> is_duplicate, linked to INC1010, never triaged fresh (cosine 0.0785 < 0.20), no Models call
dodil data sql -b "$BUCKET" "SELECT incident_id, is_duplicate, master_incident_id FROM incident_triage WHERE incident_id=1011"
#  1011 | true | 1010
 
# 6. the work note recorded the new -> triaged state change
dodil data sql -b "$BUCKET" "SELECT note_id, kind, from_state, to_state FROM incident_work_notes WHERE incident_id=1009"
#  wn-1009-triage | state_change | new | triaged
 
# 7. re-triage is idempotent — one incident_triage row per incident, never a duplicate
dodil data sql -b "$BUCKET" "SELECT count(*) AS total, count(DISTINCT incident_id) AS distinct_incidents FROM incident_triage"
#  total = 2, distinct_incidents = 2

Every one of the six bugs was found the same way, and not one of them was findable on a private bucket: components validated separately are internally consistent and still wrong together. Alone, each of the seven ITSM components passed its own ## Test — including this one. Stood up on one bucket, as the suite app actually runs, they surfaced six integration bugs in an afternoon. If you take one engineering habit from this post, take that one: validate the composition, not the parts.

One-shot

With the DODIL MCP connected, paste this to build the whole triage engine at once on your ITSM bucket:

On my DataK3 bucket `itsm` (which already has incidents/cis/services/groups + incidents.embedding and the
cmdb_impact graph), build an incident-triage engine. Confirm each step.
 
1. Create merge-keyed tables (all non-key columns nullable): incident_triage (key incident_id) with
   category, subcategory, priority, assignment_group, confidence(double), similar_incident_ids(json),
   blast_radius_json(json), rationale, model_id, is_duplicate(boolean), master_incident_id(bigint),
   triaged_at; and incident_work_notes (key note_id) with incident_id(bigint), kind, author, body,
   from_state, to_state, ts.
2. For a new incident: KNN incidents.embedding for the nearest OPEN incident — if cosine < 0.20, write
   incident_triage is_duplicate=true + master_incident_id and STOP. Else KNN the nearest RESOLVED incidents
   (fix + owning team) and graph_khop('cmdb_impact', ci_id, 5) for the blast radius.
3. Classify with kimi-k2.6 (system prompt from categories [database,network,performance,availability,
   security]) returning ONLY compact JSON {category, priority, assignment_group}; retry until content is
   non-empty. With priority_scheme=impact_urgency derive priority from an impact×urgency matrix; with
   auto_assign=true set assignment_group to the nearest resolved incident's group.
4. Write incident_triage (category/priority/assignment_group/blast_radius_json/similar_incident_ids),
   UPDATE incidents state new->triaged with the classification, and log a new->triaged incident_work_notes row.
5. Deploy an image-mode Ignite app `itsm-triage-engine` — an HTTP server (GET /healthz, POST /triage) built
   from a Dockerfile (--dockerfile-path, --port 8080, --health-path /healthz), own service account with
   k3.editor + ignite.model-user + ignite.app-developer, DODIL_SERVICE_ACCOUNT_ID = the cli- serviceAccountId
   (not the uuid), policy constants CATEGORIES/PRIORITY_SCHEME/AUTO_ASSIGN/DEDUP_THRESHOLD as env. It reads
   and writes over the Postgres wire (pg.uk-lon-1.dodil.io). Smoke-test by POSTing {incident_id} to /triage.

Ship it

itsm-triage-engine is an image-mode Ignite app — an HTTP server you POST an incident_id to on /triage. Give it a least-privilege service account (the three roles in Step 6, and set DODIL_SERVICE_ACCOUNT_ID to the cli-itsm-triage-engine-sa serviceAccountId, not the uuid), inject the policy constants as env, and deploy its Dockerfile with dodil ignite app deploy itsm-triage-engine --code ./triage-engine --dockerfile-path Dockerfile --port 8080 --health-path /healthz. The platform builds the image on deploy (Lane B) — no --runtime python, no separate build step. The full lifecycle — DODIL git → CI checks → a scanned image in the registry → versioning and rollback — is walked end to end in Ship a DODIL App.

Say the scheduling answer out loud, because the platform does not provide one. Ignite is request-invoked: an app runs when something calls it, and there is no server-side scheduler. For triage that is mostly fine — the natural trigger is your incident intake POSTing each new ticket as it arrives. But the moment any part of your ITSM needs to advance on a clock rather than on an event, you have to pick one of two patterns and commit to it:

  • an always-on pinned app — deploy with --reserved 1 --max-replicas 1 and run the loop inside the pod. Self-contained and entirely inside DODIL; you pay for a warm replica that never scales to zero.
  • an external scheduler — cron, a CI timer, any orchestrator calling the route. The app stays scale-to-zero, but your clock now depends on infrastructure outside DODIL, and its outages are yours.

Either way the caller is a service account over a platform invoke, not a browser user — which is exactly why those routes must not sit behind current_user. This matters most for the SLA clock, where "it only advances when someone loads a page" is not an SLA clock at all; itsm/sla-management works the decision through in full.

Connect your tools

Everything this build wrote lives in one DataK3 bucket, reachable by your own stack — not just the CLI. data connect itsm prints the endpoints; point your tools straight at the same rows:

You

Print the drop-in pg / bolt / grpc endpoints for the itsm bucket so I can point psql and cypher-shell at it.

DODIL MCP tools called
data_connect
Agent

pg postgresql://token:…@pg.uk-lon-1.dodil.io:5432/itsm · bolt bolt+s://bolt.uk-lon-1.dodil.io:7687 (database=itsm) · grpc table-rpc.uk-lon-1.dodil.io:443

  • SQL over Postgres wire — psql, psycopg/asyncpg (Python), node-postgres (TS).
  • Vector — pgvector (<=>) over the same wire against the same incidents.embedding rows.
  • Graph — cypher-shell or any Neo4j driver over Bolt against the cmdb_impact graph.

Full, live-validated walkthrough: Connect your tools.

The suite — seven components, one app

This package runs standalone (uvicorn routes:app) — that is what this post walks through, and the code/itsm-incident-management download is still exactly that. Deployed, the seven ITSM components compose into one app: itsm-suite-app is a single FastAPI with a router per component, one canonical models.py (23 tables) and plain imports — no importlib loader — over one bucket (itsm) and one dodil-appid pool, so seven components mean one sign-in and one bill. Fetch it as code/itsm-suite-app. It ships by the ordinary git cycle (repo → CI → registry → CD): Ship a DODIL app. One app rather than seven is the ERP default; you split only for a stated reason — a public surface against a private engine, independent scaling, a distinct trust boundary — and ITSM has none of those.

That single canonical models.py is also the structural answer to the service_id drift above. Seven routers sharing one set of table definitions cannot disagree about a column's type, because there is only one definition to disagree with.

Conclusion

Incident triage stops being a 2am gut call. The verdict is a row (incident_triage, every signal auditable), the audit trail is a row (incident_work_notes), and the judgement is a kimi-k2.6 gate over the same bucket your ITSM already lives in. A duplicate is deflected before it opens, a real incident is graded from its blast radius and routed to the team that fixed the last one — all in one Ignite app over one copy of your rows: the tickets (SQL), the similar-incident index (Vector), and the CMDB (Graph), one connection, one bill.

Next steps: