What you'll build: the major-incident bridge — the lifecycle layer of the
DODIL ITSM suite. A P1 lands on redis-session, the session cache the whole
storefront sits on. A major incident isn't a higher-priority row; it's a declared event — a commander, a
bridge line, a scoped blast radius, and a state machine you run open → mitigating → resolved. This skill
composes the pieces the rest of the suite already owns into that lifecycle:
- Declare — promote a high-priority incident to a major incident and open a bridge (
bridgesrow, a commander, a bridge URL). - Scope — pull the affected CI's TYPED blast radius from the
cmdb_impactgraph (graph_khop, filtered toimpact_rels— never a blanket traversal), and roll it up to the business services it takes down. - Track — advance the bridge
open → mitigating → resolved(each a keyed, idempotent upsert), and when it closes, the underlying incident advances toresolvedwith it.
It reads incidents/cis/services (owned by itsm/core) and the cmdb_impact graph + service_map
(owned by itsm/cmdb-blast-radius), and adds two tables of its own — bridges and bridge_events. No
model, no second copy: the bridge, the blast graph, and the ticket are one bucket, one connection.
The problem — and why it matters
When the session cache behind checkout starts evicting at peak, the cost isn't measured in tickets — it's measured in revenue per minute. A major incident with a 60-minute MTTR on a service doing $8,000/minute is a ~$480k event; shave 20 minutes off it and you've saved $160k on one incident. That's the whole reason the incident commander role exists: someone who, in the first five minutes, answers what is actually down, which customer-facing services does that take down, and who's on the bridge — and then drives the clock.
Every one of those questions is a query pillar over the same rows — and in a classic ITSM stack they live in three different systems: the ticket in Postgres, the dependency graph in a separate Neo4j CMDB, the service catalog in a warehouse, correlated by a nightly ETL. So the commander scopes a live outage against yesterday's topology, in a bridge doc pasted together by hand.
| Piece | Lands in | Pillar |
|---|---|---|
| The declared major incident + its clock | bridges (this skill) | SQL (one row per bridge) |
| The lifecycle timeline (open/mitigating/resolved) | bridge_events (this skill) | SQL (append-only) |
| The affected CI's blast radius | reads the cmdb_impact graph | Graph (graph_khop / Bolt) |
| The impacted business services | reads service_map | SQL (the rollup) |
| The ticket it's declared from | reads/advances core incidents | SQL (shared master) |
| The declare/scope loop | itsm-bridge-engine | Ignite (own service account, pure SQL/graph) |
The money is MTTR on the incidents that matter most. A bridge that turns a failing database into its full blast radius and its impacted services before the commander finishes reading the page — and tracks the clock as a queryable row, not a Slack thread — is the difference between a scoped, communicated major incident and a blind one. Here it's one bucket, two pillars, one copy of the rows.
NOTE
This is the ITSM suite's bridge lifecycle skill and it's deterministic composition — declare gate
- typed blast + service rollup + keyed upserts, no model gate. It reads what the suite already owns:
incidents/cis/servicesfromitsm/coreand thecmdb_impactgraph +service_mapfromitsm/cmdb-blast-radius. Standalone, Step 0 stubs a minimal slice of both.
Prerequisites
- The
dodilCLI (dodil auth login) and an agent connected over MCP (Claude Code, Cursor, VS Code, Codex). Every step shows an Ask your agent tab (the default — DODIL is agent-native) and a CLI tab. export BUCKET=itsm— the one bucket the whole suite shares.- itsm/core (the
incidents/cis/servicesmasters) and itsm/cmdb-blast-radius (thecmdb_impactgraph +service_map) already scaffolded. Standalone, Step 0 stubs the minimum this skill reads.
Step 0 — Stub the masters + the blast graph you consume (skip if you have the suite)
This skill scopes and tracks; it doesn't own the masters or the graph. If you built
itsm/core + itsm/cmdb-blast-radius (or
the full suite), everything below already exists — skip to Step 1. Standalone, create the three masters
this skill reads — cis, incidents, services — with every non-key column nullable:true, then the
reverse impacts edges + the cmdb_impact graph the blast scoping needs.
NOTE
Shared-type contract. incidents.opened_at / resolved_at are timestamp (not string) and
ci_id / problem_id are long — the exact types itsm/core owns. A stub that types opened_at as
a string diverges from core and breaks the sibling SLA clock's interval math. Match the types.
Create the itsm bucket, then three merge-keyed master tables with all non-key columns nullable: cis (key id long: name, ci_type, environment, owner_group, service_id, business_criticality int, status); incidents (key id long: number, short_description, ci_id long, service_id, state, priority, category, assignment_group, problem_id long, opened_at timestamp, resolved_at timestamp, resolution); services (key service_id: business_service, owner_group, tier int).
data_bucket_create→data_table_createCreated bucket itsm and 3 master tables — cis (pk id), incidents (pk id, opened_at/resolved_at timestamp, ci_id long), services (pk service_id) — all non-key columns nullable. These match the itsm/core masters; if you already ran itsm/core this is a no-op.
export BUCKET=itsm
dodil data bucket create "$BUCKET" --description "ITSM — major-incident bridge on DataK3"
dodil data table create cis -b "$BUCKET" --merge-key id \
--columns-json '[
{"name":"id","type":"long","nullable":false},
{"name":"name","type":"string","nullable":true},
{"name":"ci_type","type":"string","nullable":true},
{"name":"environment","type":"string","nullable":true},
{"name":"owner_group","type":"string","nullable":true},
{"name":"service_id","type":"string","nullable":true},
{"name":"business_criticality","type":"int","nullable":true},
{"name":"status","type":"string","nullable":true}
]'
# opened_at / resolved_at are TIMESTAMP, ci_id / problem_id are LONG (the itsm/core shared-type contract)
dodil data table create incidents -b "$BUCKET" --merge-key id \
--columns-json '[
{"name":"id","type":"long","nullable":false},
{"name":"number","type":"string","nullable":true},
{"name":"short_description","type":"string","nullable":true},
{"name":"ci_id","type":"long","nullable":true},
{"name":"service_id","type":"string","nullable":true},
{"name":"state","type":"string","nullable":true},
{"name":"priority","type":"string","nullable":true},
{"name":"category","type":"string","nullable":true},
{"name":"assignment_group","type":"string","nullable":true},
{"name":"problem_id","type":"long","nullable":true},
{"name":"opened_at","type":"timestamp","nullable":true},
{"name":"resolved_at","type":"timestamp","nullable":true},
{"name":"resolution","type":"string","nullable":true}
]'
dodil data table create services -b "$BUCKET" --merge-key service_id \
--columns-json '[
{"name":"service_id","type":"string","nullable":false},
{"name":"business_service","type":"string","nullable":true},
{"name":"owner_group","type":"string","nullable":true},
{"name":"tier","type":"int","nullable":true}
]'# models.py — the itsm/core masters this skill consumes (owned by itsm/core; stubbed here).
# Timestamps are DateTime (never string) and ci_id/problem_id are BigInteger — the shared-type
# contract. Natural PKs (the int id you assign, the service_id) — never SERIAL.
from datetime import datetime
from sqlalchemy import BigInteger, DateTime, Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
class Base(DeclarativeBase):
pass
class Ci(Base):
"""A configuration item — the graph node. Integer `id` is the graph node key (cypher() /
graph_khop take an integer-literal start node); `service_id` joins to the service catalog."""
__tablename__ = "cis"
id: Mapped[int] = mapped_column(BigInteger, primary_key=True) # natural int node key you assign
name: Mapped[str | None] = mapped_column(String, nullable=True)
ci_type: Mapped[str | None] = mapped_column(String, nullable=True) # app | service | host | database
environment: Mapped[str | None] = mapped_column(String, nullable=True)
owner_group: Mapped[str | None] = mapped_column(String, nullable=True)
service_id: Mapped[str | None] = mapped_column(String, nullable=True)
business_criticality: Mapped[int | None] = mapped_column(Integer, nullable=True)
status: Mapped[str | None] = mapped_column(String, nullable=True)
class Incident(Base):
"""The ticket a major incident is declared FROM (advanced to resolved on close). `opened_at` /
`resolved_at` are DateTime and `ci_id` / `problem_id` are BigInteger — the itsm/core
shared-type contract."""
__tablename__ = "incidents"
id: Mapped[int] = mapped_column(BigInteger, primary_key=True) # natural key
number: Mapped[str | None] = mapped_column(String, nullable=True) # e.g. "INC1006"
short_description: Mapped[str | None] = mapped_column(String, nullable=True)
ci_id: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
service_id: Mapped[str | None] = mapped_column(String, nullable=True)
state: Mapped[str | None] = mapped_column(String, nullable=True) # new|triaged|in_progress|resolved
priority: Mapped[str | None] = mapped_column(String, nullable=True) # P1..P4
category: Mapped[str | None] = mapped_column(String, nullable=True)
assignment_group: Mapped[str | None] = mapped_column(String, nullable=True)
problem_id: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
opened_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) # timestamp, not string
resolved_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) # timestamp, not string
resolution: Mapped[str | None] = mapped_column(String, nullable=True)
class Service(Base):
"""The business-service catalog — names the rollup ("Online Checkout", "Order Management")."""
__tablename__ = "services"
service_id: Mapped[str] = mapped_column(String, primary_key=True) # natural key, e.g. "svc-orders"
business_service: Mapped[str | None] = mapped_column(String, nullable=True)
owner_group: Mapped[str | None] = mapped_column(String, nullable=True)
tier: Mapped[int | None] = mapped_column(Integer, nullable=True)Now seed the storefront topology, the P1s you'll declare from, and the reverse cmdb_impact graph. This is
the same 13-CI estate the whole ITSM suite was validated on — app → host → database across four
business services, 14 TYPED ci_edges (depends_on/runs_on/part_of), the reverse impacts flip
filtered to impact_rels, and the single CREATE GRAPH (edges before the snapshot). Two of its incidents
matter here: INC1006, a P1 on redis-session (CI 2), is what you'll declare and run to resolution;
INC1009, a P1 on pg-orders-primary (CI 1), is the candidate still standing when the first bridge
closes.
In itsm, seed the 13-CI estate: pg-orders-primary 1, redis-session 2 and pg-orders-replica 11 (databases), host-app-01 7 (host), and checkout-api 3, orders-api 4, payments-gateway 5, web-storefront 6, accounts-api 8, reporting-etl 9, mobile-app-bff 10, search-index 12, cdn-edge 13 (apps); 4 business services (svc-checkout/Online Checkout, svc-orders/Order Management, svc-accounts/Customer Accounts, svc-reporting/Internal Reporting); 14 TYPED ci_edges; a service_map from cis JOIN services; a P1 open incident INC1006 on redis-session (CI 2) and a P1 INC1009 on pg-orders-primary (CI 1). Then build impacts = flip of ci_edges WHERE rel IN (depends_on,runs_on,part_of) and CREATE GRAPH cmdb_impact over cis/impacts.
data_pg→data_table_createSeeded 13 CIs, 4 business services, 14 typed ci_edges and 12 service_map rows, plus P1 INC1006 (redis-session, CI 2) and P1 INC1009 (pg-orders-primary, CI 1). Built the 14 reverse impacts edges and snapshotted CREATE GRAPH cmdb_impact — the blast graph is live.
# the impacts + service_map + ci_edges tables (itsm/cmdb-blast-radius owns these in the suite)
dodil data table create ci_edges -b "$BUCKET" --merge-key src --merge-key dst --merge-key rel \
--columns-json '[{"name":"src","type":"long","nullable":false},{"name":"dst","type":"long","nullable":false},{"name":"rel","type":"string","nullable":false}]'
dodil data table create impacts -b "$BUCKET" --merge-key src --merge-key dst \
--columns-json '[{"name":"src","type":"long","nullable":false},{"name":"dst","type":"long","nullable":false}]'
dodil data table create service_map -b "$BUCKET" --merge-key ci_id \
--columns-json '[{"name":"ci_id","type":"long","nullable":false},{"name":"service_id","type":"string","nullable":true},{"name":"business_service","type":"string","nullable":true},{"name":"tier","type":"int","nullable":true}]'
dodil data pg -b "$BUCKET" "
INSERT INTO cis (id,name,ci_type,environment,owner_group,service_id,business_criticality,status) VALUES
(1,'pg-orders-primary','database','prod','g-orders','svc-orders',1,'operational'),
(2,'redis-session','database','prod','g-platform','svc-checkout',2,'operational'),
(3,'checkout-api','app','prod','g-platform','svc-checkout',1,'operational'),
(4,'orders-api','app','prod','g-orders','svc-orders',1,'operational'),
(5,'payments-gateway','app','prod','g-platform','svc-checkout',1,'operational'),
(6,'web-storefront','app','prod','g-platform','svc-checkout',1,'operational'),
(7,'host-app-01','host','prod','g-platform',NULL,2,'operational'),
(8,'accounts-api','app','prod','g-platform','svc-accounts',2,'operational'),
(9,'reporting-etl','app','prod','g-orders','svc-reporting',3,'operational'),
(10,'mobile-app-bff','app','prod','g-platform','svc-checkout',2,'operational'),
(11,'pg-orders-replica','database','prod','g-orders','svc-orders',2,'operational'),
(12,'search-index','app','prod','g-orders','svc-orders',3,'operational'),
(13,'cdn-edge','app','prod',NULL,'svc-checkout',3,'operational')"
dodil data pg -b "$BUCKET" "
INSERT INTO services (service_id,business_service,owner_group,tier) VALUES
('svc-checkout','Online Checkout','g-platform',1),('svc-orders','Order Management','g-orders',1),
('svc-accounts','Customer Accounts','g-platform',2),('svc-reporting','Internal Reporting','g-orders',3)"
# 14 TYPED edges. src DEPENDS ON dst; failure propagates the other way (the impacts flip below).
dodil data pg -b "$BUCKET" "
INSERT INTO ci_edges (src,dst,rel) VALUES
(3,1,'depends_on'),(3,2,'depends_on'),(4,1,'depends_on'),(5,3,'depends_on'),(6,3,'depends_on'),
(6,5,'depends_on'),(8,1,'depends_on'),(9,1,'depends_on'),(11,1,'depends_on'),(12,11,'depends_on'),
(13,6,'depends_on'),(3,7,'runs_on'),(4,7,'runs_on'),(10,6,'part_of')"
# the two P1s: INC1006 on redis-session (the one we declare) and INC1009 on pg-orders-primary
dodil data pg -b "$BUCKET" "
INSERT INTO incidents (id,number,short_description,ci_id,service_id,state,priority,category,assignment_group,opened_at) VALUES
(1006,'INC1006','Session cache evictions causing random customer logouts',2,'svc-checkout','new','P1','availability','g-platform','2026-09-08 16:18:52'),
(1009,'INC1009','Order database connections exhausted, checkout failing',1,'svc-orders','triaged','P1','database','g-dba','2026-09-08 22:08:52')"
dodil data pg -b "$BUCKET" "
INSERT INTO service_map (ci_id, service_id, business_service, tier)
SELECT c.id, c.service_id, s.business_service, s.tier FROM cis c JOIN services s ON s.service_id = c.service_id"
# reverse + TYPE the edges (only impact_rels propagate failure), then snapshot the graph AFTER impacts is full
dodil data pg -b "$BUCKET" "
INSERT INTO impacts (src, dst) SELECT dst AS src, src AS dst FROM ci_edges WHERE rel IN ('depends_on','runs_on','part_of')"
dodil data pg -b "$BUCKET" "DROP GRAPH IF EXISTS cmdb_impact"
dodil data pg -b "$BUCKET" "CREATE GRAPH cmdb_impact NODES (cis KEY id) EDGES (impacts SRC src DST dst)"# models.py — the itsm/cmdb-blast-radius signal this skill consumes (owned there; stubbed here).
# ci_edges is the TYPED dependency edge table (the impact_rels knob filters it); impacts is its
# reverse flip the cmdb_impact graph is snapshotted from. Composite all-key PKs -> the upsert
# lands them ON CONFLICT DO NOTHING. service_map is the CI -> business-service rollup key.
from sqlalchemy import BigInteger, Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
class Base(DeclarativeBase):
pass
class CiEdge(Base):
"""The TYPED dependency edges (`depends_on` / `runs_on` / `part_of`). `src` depends on `dst`;
failure propagates the OTHER way (dst -> src), which the reverse `impacts` flip and the typed
recursive-CTE blast encode. The `rel` filter (impact_rels) is the typed-traversal knob."""
__tablename__ = "ci_edges"
src: Mapped[int] = mapped_column(BigInteger, primary_key=True) # the CI that depends
dst: Mapped[int] = mapped_column(BigInteger, primary_key=True) # the CI it depends on
rel: Mapped[str] = mapped_column(String, primary_key=True, default="depends_on")
class Impact(Base):
"""The reverse blast edges the `cmdb_impact` graph is built from — the flip of `ci_edges`
WHERE rel IN (impact_rels). `src` impacts `dst` (if src fails, dst is affected). CREATE GRAPH
SNAPSHOTS edges, so every impacts row must exist before the graph is (re-)created."""
__tablename__ = "impacts"
src: Mapped[int] = mapped_column(BigInteger, primary_key=True)
dst: Mapped[int] = mapped_column(BigInteger, primary_key=True)
class ServiceMap(Base):
"""The CI -> business-service rollup key — every CI joined to the service it belongs to.
The blast set JOINs here to roll up to impacted business services."""
__tablename__ = "service_map"
ci_id: Mapped[int] = mapped_column(BigInteger, primary_key=True) # natural key
service_id: Mapped[str | None] = mapped_column(String, nullable=True)
business_service: Mapped[str | None] = mapped_column(String, nullable=True)
tier: Mapped[int | None] = mapped_column(Integer, nullable=True)Step 1 — The bridge record + the event timeline
This skill owns two tables. bridges is the war-room record — one row per bridge, merge-keyed on
bridge_id, so every lifecycle transition upserts in place (never a duplicate). It carries the
commander, the state, and the scope (impacted_ci_count, impacted_services_json) and the clock stamps
(declared_at/mitigating_at/resolved_at). bridge_events is the append-only timeline — one row per
transition, keyed on event_id.
NOTE
data table create makes every non-PK column NOT NULL by default — set "nullable":true on every
optional column. A freshly-declared bridge has no mitigating_at/resolved_at yet, so those must be
nullable or the declare upsert 500s with NotNullViolation.
In the itsm bucket, create two merge-keyed tables with all non-key columns nullable. bridges (key bridge_id): incident_id(long), ci_id(long), severity, commander, state, impacted_ci_count(int), impacted_service_count(int), impacted_services_json, bridge_url, declared_at(timestamp), declared_by, mitigating_at(timestamp), resolved_at(timestamp). bridge_events (key event_id): bridge_id, incident_id(long), kind, from_state, to_state, note, author, ts(timestamp).
data_table_createCreated bridges (key bridge_id, 14 columns — record, scope, clock and the declaring user, all nullable) and bridge_events (key event_id, 9 columns). Upserts are idempotent, so each lifecycle transition updates the one bridge row.
export BUCKET=itsm
dodil data table create bridges -b "$BUCKET" --merge-key bridge_id \
--columns-json '[
{"name":"bridge_id","type":"string","nullable":false},
{"name":"incident_id","type":"long","nullable":true},
{"name":"ci_id","type":"long","nullable":true},
{"name":"severity","type":"string","nullable":true},
{"name":"commander","type":"string","nullable":true},
{"name":"state","type":"string","nullable":true},
{"name":"impacted_ci_count","type":"int","nullable":true},
{"name":"impacted_service_count","type":"int","nullable":true},
{"name":"impacted_services_json","type":"string","nullable":true},
{"name":"bridge_url","type":"string","nullable":true},
{"name":"declared_at","type":"timestamp","nullable":true},
{"name":"declared_by","type":"string","nullable":true},
{"name":"mitigating_at","type":"timestamp","nullable":true},
{"name":"resolved_at","type":"timestamp","nullable":true}
]'
dodil data table create bridge_events -b "$BUCKET" --merge-key event_id \
--columns-json '[
{"name":"event_id","type":"string","nullable":false},
{"name":"bridge_id","type":"string","nullable":true},
{"name":"incident_id","type":"long","nullable":true},
{"name":"kind","type":"string","nullable":true},
{"name":"from_state","type":"string","nullable":true},
{"name":"to_state","type":"string","nullable":true},
{"name":"note","type":"string","nullable":true},
{"name":"author","type":"string","nullable":true},
{"name":"ts","type":"timestamp","nullable":true}
]'# models.py — the two tables this skill OWNS. Natural PKs (bridge_id, event_id); the clock stamps
# (declared_at/mitigating_at/resolved_at, ts) are DateTime, never string; every non-key column is
# nullable (a freshly-declared bridge has no mitigating_at/resolved_at yet).
from datetime import datetime
from sqlalchemy import BigInteger, DateTime, Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
class Base(DeclarativeBase):
pass
class Bridge(Base):
"""The war-room record — one row per bridge, keyed on `bridge_id` (e.g. "mi-1006"), so every
lifecycle transition upserts IN PLACE (never a duplicate). It carries the commander, the
state (open|mitigating|resolved), the scope (`impacted_ci_count`, `impacted_services_json`),
and the clock stamps. A freshly-declared bridge has no `mitigating_at`/`resolved_at` yet, so
every non-key column is nullable."""
__tablename__ = "bridges"
bridge_id: Mapped[str] = mapped_column(String, primary_key=True) # natural key, e.g. "mi-1006"
incident_id: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
ci_id: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
severity: Mapped[str | None] = mapped_column(String, nullable=True)
commander: Mapped[str | None] = mapped_column(String, nullable=True)
state: Mapped[str | None] = mapped_column(String, nullable=True) # open | mitigating | resolved
impacted_ci_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
impacted_service_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
impacted_services_json: Mapped[str | None] = mapped_column(String, nullable=True)
bridge_url: Mapped[str | None] = mapped_column(String, nullable=True)
declared_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) # DateTime, not string
declared_by: Mapped[str | None] = mapped_column(String, nullable=True) # the gateway-vouched user
mitigating_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
resolved_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
class BridgeEvent(Base):
"""The append-only lifecycle timeline — one row per transition, keyed on `event_id`
(e.g. "ev-1006-1"). Upsert makes a replayed transition land the event once."""
__tablename__ = "bridge_events"
event_id: Mapped[str] = mapped_column(String, primary_key=True) # natural key
bridge_id: Mapped[str | None] = mapped_column(String, nullable=True)
incident_id: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
kind: Mapped[str | None] = mapped_column(String, nullable=True) # declared | state_change
from_state: Mapped[str | None] = mapped_column(String, nullable=True)
to_state: Mapped[str | None] = mapped_column(String, nullable=True)
note: Mapped[str | None] = mapped_column(String, nullable=True)
author: Mapped[str | None] = mapped_column(String, nullable=True)
ts: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) # DateTime, not stringStep 2 — Declare a major incident (the severity gate)
A major incident is declared, not filed. The severity_threshold param (default P1) is the gate:
an open incident (state in new/triaged/in_progress) at or above it, with no existing bridge,
is a candidate. Find it first — this is the query the declare loop runs:
In the itsm bucket, list the major-incident candidates: open incidents (state in new/triaged/in_progress) with priority P1 that don't already have a bridge — join cis for the CI name.
data_pgTwo candidates: INC1006 (P1, redis-session, CI 2) and INC1009 (P1, pg-orders-primary, CI 1) — both open, neither bridged. We declare 1006 first.
dodil data pg -b "$BUCKET" "
SELECT i.id, i.number, i.priority, i.ci_id, c.name AS ci_name
FROM incidents i JOIN cis c ON c.id = i.ci_id
WHERE i.priority = 'P1' AND i.state IN ('new','triaged','in_progress')
AND NOT EXISTS (SELECT 1 FROM bridges b WHERE b.incident_id = i.id)
ORDER BY i.id"
# 1006 | INC1006 | P1 | 2 | redis-session
# 1009 | INC1009 | P1 | 1 | pg-orders-primaryWhen auto_declare is true, open the bridge: a bridges row keyed mi-1006, state=open, a commander
(here a named IC, [email protected]; absent one, commander_default), and a declared event. The blast scope
comes next (Steps 3–5) and folds into this same row.
That commander column is the first place the gateway's work shows up in the data. POST /declare is one of
the four routes in the whole ITSM module that carries a permission — itsm:major:declare — because
declaring a major incident is the one action here whose blast radius is people, not rows: it pages the
org. bridges.declared_by and the event author are only worth anything because the identity was verified
at the edge before the app ever saw it. The Auth section
below has the full audit.
In the itsm bucket, declare a major incident for INC1006: upsert a bridge mi-1006 (incident_id 1006, ci_id 2, severity SEV1, commander [email protected], state open, bridge_url https://bridge.corp.io/mi-1006, declared_at now), and log a declared -> open bridge_events row ev-1006-1.
data_table_upsertOpened bridge mi-1006 (state=open, commander [email protected]) for INC1006 and logged the declared event. wal_written: true.
dodil data table upsert bridges -b "$BUCKET" \
--row '{"bridge_id":"mi-1006","incident_id":1006,"ci_id":2,"severity":"SEV1","commander":"[email protected]","state":"open","bridge_url":"https://bridge.corp.io/mi-1006","declared_at":"2026-09-08 22:30:57"}'
dodil data table upsert bridge_events -b "$BUCKET" \
--row '{"event_id":"ev-1006-1","bridge_id":"mi-1006","incident_id":1006,"kind":"declared","to_state":"open","note":"Major incident declared on INC1006 (P1).","author":"[email protected]","ts":"2026-09-08 22:30:57"}'Step 3 — Scope the blast radius (Graph — TYPED)
The commander's first question — what else is about to fail? — is a graph traversal.
graph_khop('cmdb_impact', 2, 5) walks the reverse impact edges forward from redis-session (CI 2) and
returns every transitively-dependent CI, hop-ranked; join cis in the same statement for names. A
failing session cache takes down five CIs across three hops:
In the itsm bucket, give me the full blast radius of redis-session (CI 2) over cmdb_impact — every impacted CI, hop-ranked, with name, type, and owner group, up to 5 hops.
data_pgBlast radius of redis-session — 5 CIs, max hop 3: hop 1 checkout-api; hop 2 payments-gateway and web-storefront; hop 3 mobile-app-bff and cdn-edge. Everything downstream of the session cache is a customer-facing surface.
dodil data pg -b "$BUCKET" "
SELECT k.hop_distance, c.name, c.ci_type, c.owner_group
FROM graph_khop('cmdb_impact', 2, 5) k
JOIN cis c ON c.id = k.node
ORDER BY k.hop_distance, k.node"
# 1 checkout-api app g-platform
# 2 payments-gateway app g-platform
# 2 web-storefront app g-platform
# 3 mobile-app-bff app g-platform
# 3 cdn-edge app (none)The same traversal in Cypher over Bolt — the graph plane speaks the Neo4j protocol and hands back node keys
(join cis for properties):
Same blast radius in Cypher over Bolt: from redis-session (id 2), follow impacts up to 5 hops on cmdb_impact and return the impacted nodes.
data_boltReturns nodes 3 at hop 1; 5 and 6 at hop 2; 10 and 13 at hop 3 — checkout-api, payments-gateway, web-storefront, mobile-app-bff, cdn-edge. The same five CIs the SQL path returned.
dodil data bolt -b "$BUCKET" -g cmdb_impact \
"MATCH (f)-[:impacts*1..5]->(a) WHERE id(f)=2 RETURN a"
# node hop_distance
# 3 1
# 5 2
# 6 2
# 10 3
# 13 3WARNING
This blast is TYPED — that's the whole point. cmdb_impact is the reverse graph built only from
rels in impact_rels (default depends_on/runs_on/part_of), owned by
itsm/cmdb-blast-radius. Which rels you let propagate changes the
number the commander acts on: drop part_of and this same blast falls from 5 CIs to 4, because
mobile-app-bff hangs off web-storefront by a composition edge and nothing else. Is a component of
a failing thing itself failing? That is a modelling decision, and the point is that you make it
deliberately rather than inheriting it from a blanket hop. mobile-app-bff is the estate's one
composition-only node, so it drops out of every blast that excludes part_of — including
pg-orders-primary's, where the same single edge is the difference between Online Checkout appearing
fully impacted and not. CMDB Blast Radius works that contrast through
on the whole estate. Never scope a bridge on an untyped traversal. Also: graph_khop is only
referenceable in a top-level FROM, not a subquery or CTE (that errors 42P01).
Step 4 — Roll the blast up to impacted business services
A list of CIs is for engineers; the bridge opens on business impact. JOIN the blast set to service_map
and GROUP BY business_service — graph traversal folded into one SQL aggregate:
In the itsm bucket, roll redis-session's blast radius up to business services: which business services does it hit, and how many CIs does each lose?
data_pgOne impacted business service: Online Checkout, losing all 5 CIs in the blast. Every downstream CI of the session cache belongs to the same customer-facing service — deep, but narrow.
dodil data pg -b "$BUCKET" "
SELECT sm.business_service, count(*) AS impacted_cis
FROM graph_khop('cmdb_impact', 2, 5) k
JOIN service_map sm ON sm.ci_id = k.node
GROUP BY sm.business_service
ORDER BY impacted_cis DESC, sm.business_service"
# Online Checkout 5Five CIs, one service is the sentence the commander opens the bridge with, and it is more useful than
either number alone. This outage is deep but narrow — it goes three hops down but never leaves Online
Checkout, so there is exactly one business owner to call and one status page to update. Contrast the
estate's other P1, INC1009 on pg-orders-primary, whose bridge scoped 9 CIs across 4 business
services — Online Checkout, Order Management, Customer Accounts and Internal Reporting all at once. Same
query, same graph; a fundamentally different incident, and you know which is which in the first minute
instead of the fortieth.
(That 9 is what mi-1009 recorded at declare time, and it is worth noticing why the number is stored
rather than recomputed on read. The estate kept changing after the bridge opened — a cdn-edge dependency
landed minutes later — so re-running the traversal today returns a larger blast. A bridge's scope is a
statement about the moment it was declared, which is exactly what an incident review needs; the live
graph answers "what is true now", and those are different questions.)
Step 5 — Fold the scope into the open bridge
Now the bridge carries its scope. Upsert mi-1006 with impacted_ci_count (blast set size),
impacted_service_count, and impacted_services_json (the distinct business services). Keyed on
bridge_id, so it lands in the same row the declare opened:
In the itsm bucket, fold the scope into bridge mi-1006: impacted_ci_count 5, impacted_service_count 1, impacted_services_json ["Online Checkout"]. Upsert on bridge_id.
data_table_upsertBridge mi-1006 now scoped: impacted_ci_count 5, impacted_service_count 1, impacted_services_json ["Online Checkout"]. wal_written: true.
dodil data table upsert bridges -b "$BUCKET" --merge \
--row '{"bridge_id":"mi-1006","impacted_ci_count":5,"impacted_service_count":1,"impacted_services_json":"[\"Online Checkout\"]"}'The --merge (partial-column) flag is deliberate: it writes only the scope columns and leaves everything
the declare set (commander, state, declared_at) untouched. That's the idempotent, keyed pattern every
transition below uses.
Step 6 — Run the clock: open → mitigating → resolved
The lifecycle is three keyed transitions, each a partial upsert on bridges + an append to bridge_events.
First, mitigation is underway — flip to mitigating and stamp mitigating_at:
In the itsm bucket, advance bridge mi-1006 to mitigating: partial upsert state=mitigating + mitigating_at now, and log an open -> mitigating bridge_events row ev-1006-2.
data_table_upsertBridge mi-1006 now mitigating (mitigating_at set); the scope columns (impacted_ci_count 5, impacted_services_json) are preserved by the partial merge. Logged ev-1006-2 (open -> mitigating).
dodil data table upsert bridges -b "$BUCKET" --merge \
--row '{"bridge_id":"mi-1006","state":"mitigating","mitigating_at":"2026-09-08 22:31:11"}'
dodil data table upsert bridge_events -b "$BUCKET" \
--row '{"event_id":"ev-1006-2","bridge_id":"mi-1006","incident_id":1006,"kind":"state_change","from_state":"open","to_state":"mitigating","note":"Failed session traffic over to the standby cache.","author":"[email protected]","ts":"2026-09-08 22:31:11"}'Then service is restored — close the bridge and advance the underlying incident to resolved in the
same beat (that link is the point: the bridge doesn't just close, the ticket it was declared from closes
with it):
In the itsm bucket, resolve bridge mi-1006: partial upsert state=resolved + resolved_at now, UPDATE incident 1006 to state=resolved + resolved_at + a resolution note, and log a mitigating -> resolved bridge_events row ev-1006-3.
data_table_upsert→data_pgBridge mi-1006 resolved (resolved_at set) and INC1006 advanced mitigating -> resolved with its resolution. Logged ev-1006-3 (mitigating -> resolved). The bridge and its ticket closed together.
dodil data table upsert bridges -b "$BUCKET" --merge \
--row '{"bridge_id":"mi-1006","state":"resolved","resolved_at":"2026-09-08 22:31:11"}'
dodil data pg -b "$BUCKET" "
UPDATE incidents
SET state='resolved', resolved_at='2026-09-08 22:31:11',
resolution='Upgraded redis-session memory limit; evictions stopped.'
WHERE id=1006"
dodil data table upsert bridge_events -b "$BUCKET" \
--row '{"event_id":"ev-1006-3","bridge_id":"mi-1006","incident_id":1006,"kind":"state_change","from_state":"mitigating","to_state":"resolved","note":"Service restored; bridge closed. Underlying INC1006 advanced to resolved.","author":"[email protected]","ts":"2026-09-08 22:31:11"}'The closed bridge is now a JOIN — the record, its scope, its clock, and the ticket it resolved, all one query:
In the itsm bucket, show bridge mi-1006 joined to its incident: bridge state, impacted_ci_count, impacted_service_count, resolved_at, and the incident's state + resolved_at.
data_pgmi-1006 -> resolved, impacted_ci_count 5, impacted_service_count 1, resolved_at 2026-09-08 22:31:11; incident 1006 -> resolved, resolved_at 2026-09-08 22:31:11. The bridge and its ticket closed together.
dodil data pg -b "$BUCKET" "
SELECT b.bridge_id, b.state AS bridge_state, b.impacted_ci_count, b.impacted_service_count,
b.resolved_at AS bridge_resolved, i.state AS incident_state, i.resolved_at AS incident_resolved
FROM bridges b JOIN incidents i ON i.id = b.incident_id
WHERE b.bridge_id = 'mi-1006'"
# mi-1006 | resolved | 5 | 1 | 2026-09-08 22:31:11 | resolved | 2026-09-08 22:31:11That last UPDATE is the most consequential line in the module
It looks like the smallest statement on the page. It is not. POST /bridges/{id}/state is the only route
in the entire ITSM module that writes incidents.state='resolved' — and resolved is what stops the
SLA clock. So the permission on this route, itsm:incident:resolve, is not "may close a ticket". It is
may stop the measurement every SLA report in the company is computed from. A component that can silently
end a measurement is a component that needs a gate, and that is the whole argument for this one. It is
deliberately a separate permission from itsm:major:declare: the person who is trusted to convene a
bridge is not automatically the person who is trusted to call it over.
The same insight shows up a second time, as a bug. Run the bridge and the SLA clock on one bucket — as the suite app actually does — and closing INC1006 here immediately becomes another component's input, with no sync job in between. That is the good news and it was also, briefly, the bad news:
WARNING
The bridge resolved INC1006 and the SLA clock didn't notice. Long after this UPDATE landed, the
clock row for 1006 still read state=in_progress, resolve_breached=true, escalation_level=2 — because the
clock's evaluation loop only walks open incidents, so a ticket that closes leaves its clock frozen at
its last open reading, forever. And the manager kept getting paged: the on-call rollup grouped
unnotified breaches with no reference to whether the ticket was still open, so INC1006's level-2 breach sat
on [email protected]'s list after the bridge had stood down. Both are fixed on the SLA side — tick
now has a retire pass, and /sla/oncall joins incidents. The sla_breaches row itself is
deliberately untouched: a breach that happened stays on the record, because that table is an audit log,
not a worklist. Full story and the fix: SLA management.
Neither was findable while SLA management owned its own bucket — nothing else there ever closed a ticket.
That is the lesson this component is best placed to teach: writing a row that another component's state
machine depends on is an integration, even when it looks like a single UPDATE. Note that the audit and
the bug arrived at the same place from opposite directions — the one route that writes state='resolved' is
exactly the route worth gating, and exactly the route whose write another component was silently missing.
These two joined six integration bugs found the same way when all seven ITSM components were finally
stood up together on one bucket on 2026-09-08. Every one of them had hidden while its component ran
alone, because alone each component was perfectly self-consistent and passed its own ## Test. The general
lesson is worth more than any individual bug: components validated separately are internally consistent
and still wrong together.
And the payoff, in the same breath: the next SLA tick after this bridge closed read
evaluated 5, escalations 0, retired 1. One incident left the open population, one clock retired, one
manager's page cleared — with no export, no connector and no nightly job, because the bridge and the SLA
engine are reading and writing the same rows in the same bucket. That is what "one copy" buys you, and
it is why the two components had to be validated together to be trusted apart.
How the pillars map
One bucket, two pillars, one copy of the rows — the bridge reaches across all of it, no second system to sync.
| Job | The usual stack | On DataK3 |
|---|---|---|
| The declared major incident + its clock | An incident tool + a bridge doc/Slack thread | bridges — a merge-keyed row, re-upserted per transition |
| The lifecycle timeline | Scattered across chat + audit logs | bridge_events — append-only, one row per transition |
| "What else is about to fail?" | Neo4j (a separate CMDB) + a nightly ETL | the cmdb_impact graph — graph_khop, Cypher over Bolt, same bucket |
| "Which business services does it take down?" | Graph traversal + a warehouse JOIN | one GROUP BY over service_map, graph folded in |
| Closing the ticket with the bridge | A cross-system sync | one UPDATE incidents on the shared master |
| The declare/scope loop | A workflow runtime + connectors | Ignite itsm-bridge-engine — scale-to-zero, pure SQL/graph |
No model in the loop, no second copy, no drift between the bridge and the ticket — the blast radius, the impacted services, and the incident it closes are the same live rows, one connection.
Routes
The download (see Get the code) fronts this bucket with a small FastAPI app, routes.py — bridge
CRUD plus the deterministic composition that runs the war room: the declare gate, the TYPED blast (graph),
the service rollup, and the open → mitigating → resolved lifecycle. This is what the itsm-bridge-engine
serves. Every route follows the same DataK3 rules the package bakes in, so quoting it is documenting them.
The connection and the one write helper live in db.py — a DataK3 bucket is a Postgres endpoint
(db name = the bucket, user = the literal token, password = your DODIL token), so there's no data connect
step in code. upsert() is the only writer every route uses — INSERT … ON CONFLICT DO UPDATE, because on
DataK3 a bare re-INSERT of an already-committed key raises duplicate-key 23505; upsert makes a retry or
a replay land the row once (one bridges row per bridge_id, ever).
The declare gate + declare (SQL). GET /candidates is the query the loop runs — OPEN incidents at or
above the threshold with no existing bridge. POST /declare {incident_id} verifies the candidate, opens the
keyed bridges row mi-<id> (state=open, the commander, severity from priority), and logs the declared
event:
# routes.py — the declare gate + declare (open the bridge)
# GATED: declaring a major incident pages the org, so the route needs the permission — and the
# gateway-vouched user it yields is what lands in bridges.declared_by.
@router.post("/declare")
def declare(d: DeclareIn, user=Depends(require_permission("itsm:major:declare")),
s: Session = Depends(db)):
inc = s.get(Incident, d.incident_id)
if not inc:
raise HTTPException(404, "no such incident")
if (inc.state not in OPEN_STATES) or (
PRIORITY_RANK.get(inc.priority or "", 99) > PRIORITY_RANK.get(SEVERITY_THRESHOLD, 1)):
raise HTTPException(409, f"incident {d.incident_id} is not an open candidate "
f"at/above {SEVERITY_THRESHOLD} (state={inc.state}, priority={inc.priority})")
bridge_id = f"mi-{d.incident_id}"
commander = d.commander or COMMANDER_DEFAULT
now = _now()
upsert(s, Bridge, [{
"bridge_id": bridge_id, "incident_id": inc.id, "ci_id": inc.ci_id,
"severity": SEVERITY_OF.get(inc.priority or "", "SEV1"), "commander": commander,
"state": "open", "bridge_url": f"{BRIDGE_URL_BASE}/{bridge_id}", "declared_at": now,
}], key="bridge_id")
upsert(s, BridgeEvent, [{
"event_id": f"ev-{d.incident_id}-1", "bridge_id": bridge_id, "incident_id": inc.id,
"kind": "declared", "to_state": "open",
"note": f"Major incident declared on {inc.number} ({inc.priority}).",
"author": commander, "ts": now,
}], key="event_id")
s.commit()
return {"ok": True, "bridge_id": bridge_id, "incident_id": inc.id, "ci_id": inc.ci_id,
"state": "open", "commander": commander}Live-verified: POST /declare {incident_id: 1006} opened mi-1006 (state=open, SEV1, commander
[email protected], declared_by stamped from the injected identity, wal_written: true), and the post-close
GET /candidates correctly surfaced only INC1009 (1006 excluded — bridged and resolved), which was then
declared in turn as mi-1009.
Scope the blast (GRAPH — two paths). The commander's first question is a graph traversal. The package
carries both graph rules the DataK3 tables engine enforces. The default path traverses the pre-built
cmdb_impact graph with cypher() as a top-level SELECT (never a subquery/CTE), integer-literal anchor,
and feeds the node ids into a SQL IN (…) (DuckDB has no = ANY(array)). Do not alias inside cypher()
— RETURN a AS node errors 42601; the projection column is already node:
# routes.py — the untyped blast over the pre-built typed graph (cypher, top-level SELECT)
def _graph_blast(s: Session, ci_id: int, max_hops: int = MAX_HOPS) -> list[int]:
rows = s.execute(text(
"SELECT node FROM cypher('" + GRAPH + "', "
"'MATCH (f)-[*1.." + str(int(max_hops)) + "]->(a) "
"WHERE id(f) = " + str(int(ci_id)) + " RETURN a')"
)).scalars().all()
return [int(n) for n in dict.fromkeys(rows) if int(n) != int(ci_id)]The cypher() subset can't filter on an edge's rel, so the typed blast (the impact_rels knob, at
query time, no graph rebuild) uses a recursive CTE over ci_edges with an explicit rel IN (…) filter —
this is what makes the typed-narrowing proof (redis-session 5 → 4 CIs) a one-parameter change:
# routes.py — the TYPED blast off the edge table (recursive CTE, rel IN (impact_rels))
def _typed_blast(s: Session, ci_id: int, impact_rels: list[str] = None,
max_hops: int = MAX_HOPS) -> list[int]:
rels = impact_rels if impact_rels is not None else IMPACT_RELS
rel_in = ",".join("'" + r.replace("'", "") + "'" for r in rels)
fam = s.execute(text(
"WITH RECURSIVE blast(node) AS ("
" SELECT " + str(int(ci_id)) + " "
" UNION "
" SELECT e.src FROM ci_edges e JOIN blast b ON e.dst = b.node "
" WHERE e.rel IN (" + rel_in + ")) "
"SELECT node FROM blast"
)).scalars().all()
return [int(n) for n in dict.fromkeys(fam) if int(n) != int(ci_id)]GET /incidents/{incident_id}/blast (add ?typed=true for the recursive-CTE path) hydrates the blast set
from cis via that IN (…), then rolls it up to business services through service_map. Live-verified for
redis-session (CI 2): both paths return checkout-api, payments-gateway, web-storefront,
mobile-app-bff, cdn-edge (5 CIs, max hop 3) rolling up to Online Checkout (5); over Bolt the same
walk returns nodes 3, 5, 6, 10, 13 at hops 1/2/2/3/3. Dropping part_of from impact_rels narrows the
typed path to 4 — mobile-app-bff leaves.
Fold the scope + run the clock. POST /bridges/{bridge_id}/scope upserts impacted_ci_count,
impacted_service_count, and impacted_services_json onto the same bridge row (keyed on bridge_id).
POST /bridges/{bridge_id}/state {state} is the lifecycle transition — a partial-column merge sets state +
the matching clock stamp, leaves the scope columns untouched, logs a state_change event, and on resolved
advances the underlying incident (a fresh statement after the bridge commit — DataK3 has no read-your-writes
inside an open transaction):
# routes.py — advance the lifecycle; on resolve, advance the underlying incident too
# GATED: this is the ONLY route in the module that writes incidents.state='resolved' — the write
# that stops the SLA clock. A separate permission from declare, on purpose.
@router.post("/bridges/{bridge_id}/state")
def advance_state(bridge_id: str, t: StateIn,
user=Depends(require_permission("itsm:incident:resolve")),
s: Session = Depends(db)):
if t.state not in ("mitigating", "resolved"):
raise HTTPException(422, "state must be 'mitigating' or 'resolved'")
br = s.get(Bridge, bridge_id)
if not br:
raise HTTPException(404, "no such bridge")
from_state = br.state
now = _now()
merged = {c.name: getattr(br, c.name) for c in Bridge.__table__.columns}
merged["state"] = t.state
merged[f"{t.state}_at"] = now
upsert(s, Bridge, [merged], key="bridge_id")
seq = 2 if t.state == "mitigating" else 3
upsert(s, BridgeEvent, [{
"event_id": f"ev-{br.incident_id}-{seq}", "bridge_id": bridge_id,
"incident_id": br.incident_id, "kind": "state_change", "from_state": from_state,
"to_state": t.state, "note": t.note or f"Bridge advanced {from_state} -> {t.state}.",
"author": t.author or br.commander, "ts": now,
}], key="event_id")
s.commit()
if t.state == "resolved" and br.incident_id is not None:
s.execute(text(
"UPDATE incidents SET state='resolved', resolved_at=:ra, resolution=:res WHERE id=:id"
), {"ra": now, "res": t.resolution or
f"Resolved on major-incident bridge {bridge_id}.", "id": br.incident_id})
s.commit()
return {"ok": True, "bridge_id": bridge_id, "from_state": from_state, "to_state": t.state,
"incident_resolved": t.state == "resolved"}Live-verified end-to-end: open → mitigating preserved the scope columns (impacted_ci_count stayed 5
through the flip), → resolved set resolved_at and advanced INC1006 → resolved with its
resolution, 3 bridge_events rows were logged, and a re-declare of mi-1006 left bridges at one row per
bridge_id. Adding a new operation touches only routes.py (and maybe models.py) — write via
upsert, graph via cypher(…) or the recursive CTE (see EXTENDING.md).
Note what these two routes carry that the rest of the file does not: POST /declare takes
Depends(require_permission("itsm:major:declare")) and POST /bridges/{id}/state takes
Depends(require_permission("itsm:incident:resolve")). Two of the four permissions in the entire ITSM
module live in this one file — not because major incidents are important-sounding, but because these are
the two routes whose effects escape the database: one pages the org, the other stops the SLA clock.
Everything else here — the candidate list, the blast query, the service rollup, the scope fold — takes
Depends(current_user) and nothing more.
Auth — config at the edge, a role gate in the app
End-user login on Ignite is configuration, not code. The app deploys with the itsm-suite dodil-appid
pool attached (user_pool: itsm-suite in .dodil/deploy.yaml), and the per-cluster Ignite gateway
runs the entire browser login at the edge — PKCE S256 against the pool issuer, an AEAD-sealed host-only
session cookie, single-flight refresh, EdDSA JWT verification against trust anchors this app does not hold.
It then injects the verified identity into every request it forwards:
X-Dodil-User—sub,email,connection,app_roles(plain JSON)X-Dodil-User-Jwt— the raw verified pool token, carrying the catalog-expandedpermissionsclaimX-Dodil-Auth-Source—pool(an app end-user) orplatform(an operator or service-account invoke)
Any inbound copy of those headers is stripped first, on every mode and every principal, so a caller can never forge them.
What survives in the package is a small auth.py that ships no verifier — no JWKS client, no
issuer/audience env, no crypto dependency, and no pyjwt in requirements.txt. It reads the injected
header and keeps the one job the app still owns: role-based gating.
# auth.py — a header-trust reader, not a verifier (the gateway already checked the JWT)
def current_user(request: Request) -> dict:
raw = request.headers.get("x-dodil-user") # {"sub","email","connection","app_roles"}
... # + permissions read off x-dodil-user-jwt
raise HTTPException(401, "end-user login required — no X-Dodil-User from the gateway")
def require_permission(perm: str):
"""Gate a route on a pool permission: Depends(require_permission("itsm:major:declare"))."""
def _dep(user: dict = Depends(current_user)) -> dict:
if perm not in user["permissions"] and perm not in user["roles"]:
raise HTTPException(403, f"missing permission: {perm}")
return user
return _depThe gate audit — four permissions across seven components
ITSM uses namespaced permissions, <module>:<object>:<verb>, so one customer pool can carry every ERP
module's roles without collision (itsm:change:approve is not crm:change:approve). Across all seven
components the audit left exactly four gates standing — and two of them are in this one package:
| Permission | Gates | Why this one and not the rest |
|---|---|---|
itsm:change:approve | change-management: POST /assess, POST /changes/{id}/transition, POST /policies | accepting the risk of a production change. change_approvals.decided_by records the gateway-vouched user |
itsm:major:declare | major-incident: POST /declare | declaring a major incident pages the org — the one action here whose blast radius is people, not rows. bridges.declared_by records who |
itsm:incident:resolve | major-incident: POST /bridges/{id}/state | the only route in the module that writes incidents.state='resolved' — the write that stops the SLA clock |
itsm:cmdb:rebuild | cmdb-blast-radius: POST /graph/assemble | it TRUNCATEs impacts + service_map and DROP+CREATEs both graphs; every other component's blast radius comes from what it leaves behind |
(Those are the standalone paths this package serves. Mounted in the suite app each router sits under its own
prefix — /major-incident/declare, /change-management/assess, and so on.)
That this component holds half the module's permissions is not seniority, it is consequence. Both of its gated routes have effects that leave the database: one pages humans, the other ends a measurement. And they are deliberately two permissions rather than one — convening a bridge and declaring it over are different acts of judgement, and an org that wants them held by different people can express that.
Four of seven components have zero gates — on purpose
itsm-core, itsm-incident-management, itsm-problem-management and itsm-sla-management carry no
permission at all. CMDB CRUD, triage and clustering are the service desk's ordinary work, and a
permission that every agent on the desk must hold is ceremony — which is exactly what an auditor discounts.
The audit deleted three gates that earlier versions of these posts described (incidents:triage,
problems:write, and cmdb:write on an idempotent per-CI precompute) and added one the roadmap had not
predicted (itsm:cmdb:rebuild), because the route it guards is the most destructive operation in the
module. Fewer gates, each defensible, beats a gate per verb.
The pool is created once for the whole suite, with the role catalog those gates check:
Create a dodil-appid pool itsm-suite with email+password, and set its role catalog: a service-desk agent needs no special permission; a change manager may approve changes; an incident commander may declare a major incident and resolve it; a CMDB admin may rebuild the graph.
Pool itsm-suite created — issuer https://appid.dodil.io/ihdiash/itsm-suite, audience pool:itsm-suite, email+password (local) enabled. Catalog set: agent holds no permissions (the ungated service-desk work); change-manager = itsm:change:approve; incident-commander = itsm:major:declare + itsm:incident:resolve; cmdb-admin = itsm:cmdb:rebuild. A user's next token carries app_roles plus the catalog-expanded permissions claim the gates read.
dodil appid pool create itsm-suite --with-local
# issuer: https://appid.dodil.io/ihdiash/itsm-suite audience: pool:itsm-suite
dodil appid roles set itsm-suite \
agent= \
change-manager=itsm:change:approve \
incident-commander=itsm:major:declare,itsm:incident:resolve \
cmdb-admin=itsm:cmdb:rebuildagent holds no permissions, and that is the honest shape of a service desk: most of the module is
ungated work done by everyone, with a short list of consequential actions held by a few.
The two-plane rule is unchanged: the pool identifies the user; the app still reaches DataK3 through
its own service account (sa_token.py mints and refreshes a client_credentials token for the pg-wire
password) — an app-user is never a bucket principal. Locally, with no gateway in front of
uvicorn routes:app, opt in to a stub identity with DEV_ALLOW_ANON=1; the stub carries no permissions
unless you grant them (DEV_USER_PERMISSIONS=itsm:major:declare,itsm:incident:resolve,…), so the gated
routes stay gated on a laptop too. That is visible in the validation data: mi-1006 and mi-1009 both
carry declared_by = oncall@localhost, the stub identity that held itsm:major:declare for the run.
Today the pool is email+password (local); oauth/oidc/saml corporate SSO switch on per pool later,
with no app change — the app never sees a token either way. Full flows: App
authentication and App roles.
The one route with no identity at all
POST /sla/tick— the sibling SLA engine's recompute — carries no identity dependency whatsoever: not a permission, not evenDepends(current_user). It is a machine heartbeat, called by a service account over a platform invoke, which carries noX-Dodil-Userat all. Acurrent_userdependency there would401the clock; the engine would stop and every incident's breach flags would silently go stale — the failure mode of an SLA system that reports green. Exposure is the ingress's job (public_invoke=false), not a user permission. This applies to every recompute route in every module — a rollup, a re-materialization, an embedding backfill: gate the door, not the caller.
Get the code
The package is a real download — code/itsm-major-incident/v1.tar. This
post is a walkthrough of exactly those files; the tarball is source only (no Dockerfile — deploy is in
Ship it):
models.py # SQLAlchemy — bridges, bridge_events (owned) + cis/incidents/services + ci_edges/impacts/service_map (consumed)
routes.py # FastAPI — bridge CRUD + declare gate + TYPED blast (graph) + service rollup + open→mitigating→resolved
db.py # the lazy engine + the ON CONFLICT upsert helper every route uses
sa_token.py # mints/refreshes the service-account client_credentials token (the pg-wire password)
auth.py # header-trust role gate — reads what the gateway injected. No verifier, no JWKS
.env.example # BUCKET + PG_HOST/PG_PORT + DODIL_TOKEN + the skill params (IMPACT_RELS, MAX_HOPS, …)
requirements.txt # sqlalchemy, psycopg[binary], pgvector, fastapi, uvicorn, pydantic, httpx (no pyjwt)
README.md # what it is and how to run it
EXTENDING.md # the pattern for adding a workflow route
PLATFORM.md # the DataK3/Ignite invariants — every line a scar from a real failure
Two of those are worth calling out. sa_token.py is lazy on purpose: it mints no token at import, so
app.openapi() builds with no credentials at all and CI can generate an API client without secrets.
PLATFORM.md ships the platform rules inside the tar, so whoever downloads the package gets them
with the code rather than having to find them in a repo they can't see. And note what is not in
.env.example any more: no APPID_ISSUER, no APPID_AUDIENCE, no JWKS URL. There is nothing to configure
— the gateway does the login. What it does carry, commented out, is the local-dev block
(DEV_ALLOW_ANON=1, DEV_USER_SUB, DEV_USER_EMAIL, DEV_USER_PERMISSIONS) for running without a
gateway in front.
Run it — point .env at your bucket, create the tables from the models, serve:
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # then set DODIL_TOKEN (your `dodil auth login` token)
# BUCKET defaults to "itsm"; create it once (Step 0), then create the tables from the models:
python -c "import db, models; models.Base.metadata.create_all(db.engine)"
uvicorn routes:app --reload
# GET /candidates · POST /declare {incident_id} · GET /incidents/{id}/blast[?typed=true]
# POST /bridges/{id}/scope · POST /bridges/{id}/state {state:"mitigating"|"resolved"}Customize — the decisions this skill asks you
Q1 · severity_threshold — what becomes a major incident?
"At or above which priority does an OPEN incident become a major-incident candidate?" → Default P1. The declare gate scans open incidents (
new/triaged/in_progress) at or above this priority with no existing bridge. Raise it to reserve the bridge for the true criticals; lower it to P2 to war-room more aggressively.
Q2 · auto_declare — open the bridge, or surface the candidates?
- true (default) → a candidate at/above
severity_thresholdis opened as a bridge automatically (state=open,commander=commander_default). - false → the declare step returns the candidate list only; a human opens the bridge. The rest of the lifecycle (scope, track, resolve) is unchanged.
Q3 · commander_default — who commands until an IC takes over?
"Who commands a freshly-declared bridge until a named IC takes over?" → Default
major-incident-manager, written tobridges.commanderat declare time. Reassign per bridge with a partial upsert (--mergeonbridge_id).
Q4 · impact_rels — which relationships propagate failure? (shared)
"Which relationship kinds count as failure propagation —
depends_on,runs_on,part_of?" → The TYPED-traversal knob, shared withitsm/cmdb-blast-radius(which builds thecmdb_impactgraph from exactly these rels). Default all three. Droppart_ofwhen composition shouldn't propagate failure — the blast, and therefore the bridge'simpacted_ci_count, shrinks. A blanket traversal over-counts.
Q5 · max_hops — how deep does a blast reach? (shared)
→ The depth of graph_khop('cmdb_impact', <ci>, max_hops) and the Bolt *1..N bound. Default 5 covers
the estate's app → host → database depth (redis-session's blast bottoms out at hop 3).
(Suite-shared answers — bucket, impact_rels, max_hops — are asked once at the suite level and not
re-asked here.)
Test
Every query below ran live against DataK3 on 2026-09-08 (org IHDIASH, bucket itsm) — and, this
time, not on a bucket of its own. All seven ITSM components were stood up together on that one bucket,
which is the only reason the two clock findings above were visible at all. Real results are inline.
tested_branches: full (impact_rels: [depends_on, runs_on, part_of], the whole
open → mitigating → resolved lifecycle) and typed-narrowing (impact_rels: [depends_on, runs_on]).
# 1) declare gate — the open P1s with no bridge are the candidates
dodil data pg -b "$BUCKET" "
SELECT i.id FROM incidents i WHERE i.priority='P1' AND i.state IN ('new','triaged','in_progress')
AND NOT EXISTS (SELECT 1 FROM bridges b WHERE b.incident_id=i.id)" # -> 1006, 1009
# 2) typed blast of redis-session (CI 2) = 5 CIs, max hop 3
dodil data pg -b "$BUCKET" "
SELECT k.hop_distance, c.name FROM graph_khop('cmdb_impact', 2, 5) k
JOIN cis c ON c.id = k.node ORDER BY k.hop_distance, k.node"
# -> 1 checkout-api ; 2 payments-gateway ; 2 web-storefront ; 3 mobile-app-bff ; 3 cdn-edge
# 3) same over Bolt -> nodes 3, 5, 6, 10, 13
dodil data bolt -b "$BUCKET" -g cmdb_impact "MATCH (f)-[:impacts*1..5]->(a) WHERE id(f)=2 RETURN a"
# 4) service rollup -> Online Checkout (5); the bridge folds impacted_ci_count=5, service_count=1
dodil data pg -b "$BUCKET" "
SELECT sm.business_service, count(*) FROM graph_khop('cmdb_impact', 2, 5) k
JOIN service_map sm ON sm.ci_id = k.node GROUP BY sm.business_service ORDER BY 2 DESC"
# -> Online Checkout | 5
# 5) lifecycle -> bridge resolved AND the underlying incident resolved, together
dodil data pg -b "$BUCKET" "
SELECT b.state, b.resolved_at, i.state AS incident_state, i.resolved_at AS incident_resolved
FROM bridges b JOIN incidents i ON i.id=b.incident_id WHERE b.bridge_id='mi-1006'"
# -> resolved | 2026-09-08 22:31:11 | resolved | 2026-09-08 22:31:11 (3 bridge_events: declared, ->mitigating, ->resolved)
# 6) TYPED proof: drop part_of -> redis-session's blast shrinks 5 -> 4
# (mobile-app-bff hangs off web-storefront by a part_of edge and nothing else)
dodil data pg -b "$BUCKET" "
WITH RECURSIVE blast(node) AS (
SELECT 2 UNION
SELECT e.src FROM ci_edges e JOIN blast b ON e.dst = b.node
WHERE e.rel IN ('depends_on','runs_on'))
SELECT count(*) - 1 AS typed_blast FROM blast" # -> 4 (5 with part_of included)
# 7) the SEAM: the tick right after this bridge closed retires the clock it froze
# POST /sla/tick -> {"evaluated":5,"escalations":0,"retired":1}
dodil data pg -b "$BUCKET" "SELECT state, resolve_breached, escalation_level FROM incident_sla WHERE incident_id=1006"
# -> resolved | true | 0 (breached judged at the moment of resolution; escalation zeroed)
# 8) idempotent: re-declare the same incident -> one row per bridge_id
dodil data pg -b "$BUCKET" "SELECT count(*) AS total, count(DISTINCT bridge_id) AS distinct_bridges FROM bridges"Live-captured values: the declare gate returned candidates INC1006 + INC1009;
graph_khop('cmdb_impact', 2, 5) returned checkout-api / payments-gateway / web-storefront /
mobile-app-bff / cdn-edge at hops 1/2/2/3/3, and the Bolt MATCH returned nodes 3, 5, 6, 10, 13; the
service rollup returned Online Checkout (5) — one service, five CIs; the bridge upserts returned
{wal_ulid, wal_written: true} with declared_by stamped from the injected identity; the partial --merge
transitions preserved the scope columns (impacted_ci_count stayed 5 through the mitigating flip);
and the close advanced INC1006 → resolved alongside the bridge (3 bridge_events rows: declared→open,
open→mitigating, mitigating→resolved). The typed proof showed the blast at 5 CIs with all three rels
and 4 with part_of dropped. Re-declaring left bridges at one row per bridge_id, and the
post-close GET /candidates correctly surfaced INC1009 — which was then declared as mi-1009,
recording a scope of 9 CIs across 4 business services off pg-orders-primary, a visibly different shape
of incident from the same query. Finally the cross-component assertion: the next POST /sla/tick read
evaluated 5, escalations 0, retired 1, and INC1006's clock row now reads
state=resolved, escalation_level 0, while its sla_breaches row survives untouched as an audit record.
One-shot
With the DODIL MCP connected, paste this to run the whole major-incident bridge on your ITSM bucket — it stubs a minimal core + blast slice so it runs standalone:
On my DataK3 bucket `itsm` (SQL + graph), run a major-incident bridge. Confirm each step.
1. If itsm/core + itsm/cmdb-blast-radius are absent, stub them: cis (key id) with the 13-CI storefront estate,
services (key service_id) with 4 business services, incidents (key id; opened_at/resolved_at TIMESTAMP,
ci_id LONG) with a P1 open INC1006 on redis-session (CI 2) and a P1 INC1009 on pg-orders-primary (CI 1);
ci_edges (key src,dst,rel) with the 14 TYPED depends_on/runs_on/part_of edges; service_map from cis JOIN
services; impacts = flip of ci_edges WHERE rel IN (depends_on,runs_on,part_of); CREATE GRAPH cmdb_impact
over cis/impacts (edges before the snapshot).
2. Create bridges (key bridge_id) + bridge_events (key event_id), all non-key columns nullable.
3. DECLARE: find an OPEN incident at/above P1 with no bridge (INC1006) -> open bridge mi-1006 (state=open,
SEV1, a named commander, declared_by from the signed-in user) + a declared bridge_events row.
4. SCOPE: graph_khop('cmdb_impact', 2, 5) JOIN cis -> checkout-api / payments-gateway / web-storefront /
mobile-app-bff / cdn-edge (typed, not blanket). Roll up via service_map -> Online Checkout (5). Fold
impacted_ci_count=5, impacted_service_count=1, impacted_services_json into mi-1006 (partial --merge upsert).
5. TRACK: partial-upsert open -> mitigating (mitigating_at) -> resolved (resolved_at), each with a
bridge_events row; on resolve, UPDATE incident 1006 to state=resolved + resolved_at + resolution.
6. Prove typing: redis-session's blast is 5 CIs with all three rels but 4 with part_of dropped
(mobile-app-bff is reachable only over a part_of edge).
7. Prove the seam: POST /sla/tick after the close -> the retired clock for 1006 (evaluated 5, retired 1),
and confirm /sla/oncall no longer pages the manager for it while sla_breaches still holds the record.Connect your tools
Everything this build wrote lives in one DataK3 bucket, reachable by your own stack — not just the CLI.
data connect itsm prints the endpoints; point your tools straight at the same rows:
Print the drop-in pg / bolt / grpc endpoints for the itsm bucket so I can point psql and cypher-shell at it.
data_connectpg postgresql://token:…@pg.uk-lon-1.dodil.io:5432/itsm · bolt bolt+s://bolt.uk-lon-1.dodil.io:7687 (database=itsm) · grpc table-rpc.uk-lon-1.dodil.io:443
dodil data connect itsm # pg / bolt / grpc endpoints- SQL over Postgres wire —
psql,psycopg/asyncpg(Python),node-postgres(TS) — thebridges+bridge_eventsrows and the sharedincidentsmaster. - Graph over Bolt — a Neo4j driver or
cypher-shellagainst thecmdb_impactblast graph.
Full, live-validated walkthrough: Connect your tools.
Ship it
The declare/scope loop is one small image-mode Ignite app — itsm-bridge-engine. On a page or a tick it
scans for candidates at/above severity_threshold, scopes the typed blast, and opens/advances the bridge —
an HTTP server (GET /healthz, POST /declare {incident_id}, POST /bridge/{id}/state) packaged by a
Dockerfile and built on deploy, not a handler(payload, ctx) compile-mode function. Because the whole
lifecycle is deterministic composition — no Models gate — its service account needs just k3.editor
(write the tables, advance the incident) and ignite.app-developer (the deploy identity) — no
ignite.model-user, no token-billed calls. Reads/writes go over the drop-in Postgres wire
(pg.uk-lon-1.dodil.io:5432, dbname=itsm, user=token, password = the SA access token) via psycopg
(graph_khop + INSERT-is-upsert); there is no K3 HTTP API. Set DODIL_SERVICE_ACCOUNT_ID to the
cli-… serviceAccountId, not the uuid (the uuid fails client_credentials with invalid_client).
Create a service account itsm-bridge-engine-sa, grant it k3.editor plus ignite.app-developer (pure SQL/graph — no ignite.model-user), then deploy my ./bridge-engine (image mode — its Dockerfile builds on deploy) to Ignite as itsm-bridge-engine on port 8080 with health path /healthz, passing the service-account creds, BUCKET, SEVERITY_THRESHOLD, AUTO_DECLARE, COMMANDER_DEFAULT, IMPACT_RELS, and MAX_HOPS as runtime env. Use the cli- serviceAccountId as DODIL_SERVICE_ACCOUNT_ID, not the uuid. Then POST {incident_id:1006} to /declare to smoke-test.
auth_service-account_create→auth_service-account_grant-role→ignite_app_deployCreated itsm-bridge-engine-sa (serviceAccountId cli-itsm-bridge-engine-sa), granted k3.editor + ignite.app-developer, built + deployed itsm-bridge-engine (image:build, public FQDN on :8080, scale-to-zero). POST {incident_id:1006} to /declare opened bridge mi-1006 (open, impacted_ci_count 5), written=true.
# create prints the serviceAccountId (cli-itsm-bridge-engine-sa) + the secret. The client_credentials
# client_id is that serviceAccountId — NOT the uuid (the uuid fails with invalid_client).
dodil auth service-account create itsm-bridge-engine-sa
SA_ID=cli-itsm-bridge-engine-sa
# grant-role addresses the SA by its uuid; the env below uses the serviceAccountId.
SA_UUID=$(dodil auth service-account list -o json | python3 -c "import sys,json;print([s['uuid'] for s in json.load(sys.stdin) if s['serviceAccountId']=='cli-itsm-bridge-engine-sa'][0])")
dodil auth service-account grant-role "$SA_UUID" k3-authorization-service k3.editor # write tables + advance incidents
dodil auth service-account grant-role "$SA_UUID" ignite-authorization-service ignite.app-developer # deploy identity — NO ignite.model-user (no model)
# IMAGE mode — the platform builds ./bridge-engine/Dockerfile on deploy (Lane B / Kaniko build-on-deploy).
dodil ignite app deploy itsm-bridge-engine \
--code ./bridge-engine --dockerfile-path Dockerfile \
--port 8080 --health-path /healthz --allow-unauthenticated \
--env BUCKET="$BUCKET" --env SEVERITY_THRESHOLD=P1 --env AUTO_DECLARE=true \
--env COMMANDER_DEFAULT=major-incident-manager --env IMPACT_RELS=depends_on,runs_on,part_of --env MAX_HOPS=5 \
--env DODIL_SERVICE_ACCOUNT_ID="$SA_ID" --env DODIL_SERVICE_ACCOUNT_SECRET="$SA_SECRET"
# runtime image:build -> public FQDN itsm-bridge-engine-$ORG-8080.ignite.dodil.cloud
# the app is an HTTP server now — smoke-test with a POST to /declare (not ignite invoke)
curl -sS -X POST "https://itsm-bridge-engine-$ORG-8080.ignite.dodil.cloud/declare" \
-H 'Content-Type: application/json' -d '{"incident_id":1006}'NOTE
Deploy: image mode (Lane B). The bridge lifecycle above was validated live, one call at a time
(## Test); the deploy wrapper reuses the image-mode pattern validated live 2026-09-02 on the sibling
itsm-triage-engine / crm-lead-scorer engines (deploys, serves /healthz + its route unauthenticated,
writes durably). It's an HTTP server (Dockerfile + --dockerfile-path, Kaniko build-on-deploy) — not
--runtime python. There's no server-side scheduler — drive /declare from your alerting webhook, or
pin a warm poll loop with --reserved 1 --max-replicas 1. If dodil ignite app deploy returns
IAM resource registration failed: broken pipe, deploy under a fresh app name — the half-created app
can't be updated or deleted.
The full lifecycle — DODIL git → CI checks → a scanned registry image → versioning and rollback — is in Ship a DODIL App.
The scheduler question, which you have to answer
Ignite is request-invoked, and there is no server-side scheduler. For the bridge that is mostly fine — a major incident is declared by a human or an alerting webhook, both of which are requests. But the component this one hands off to has no such luxury: an SLA clock that only advances when someone loads a page is not an SLA clock, and the moment this bridge resolves a ticket, something has to tick for the clock to notice. There are exactly two honest answers, and a deployment must pick one and say which:
- an always-on pinned app —
--reserved 1 --max-replicas 1, running its own loop in the pod and callingtick()on an interval. Self-contained, and it costs you a permanently warm replica; it never scales to zero. - an external scheduler — cron, a CI timer, any orchestrator POSTing
/sla/tick. The app stays scale-to-zero, but your clock now depends on infrastructure outside DODIL.
Either way the caller is a service account over a platform invoke, not a browser user — which is exactly
why /sla/tick must not sit behind current_user. This is a real platform gap, it is the most
buyer-visible one in the module, and it is discussed in full in
SLA management.
The suite — seven components, one app
This package runs standalone (uvicorn routes:app) — that is what this post walks through, and the
code/itsm-major-incident download is still exactly that. Deployed, the
seven ITSM components compose into one app: itsm-suite-app is a single FastAPI with a router per
component, one canonical models.py (23 tables) and plain imports — no importlib loader — over one
bucket (itsm) and one dodil-appid pool, so seven components mean one sign-in and one bill. Fetch it as
code/itsm-suite-app. It ships by the ordinary git cycle (repo → CI → registry →
CD): Ship a DODIL app.
One app rather than seven is the ERP default; you split only for a stated reason — a public surface against a private engine, independent scaling, a distinct trust boundary — and ITSM has none of those. It is also, as this post has shown twice over, the configuration in which the module is actually correct: the bridge closing a ticket and the SLA clock retiring it are one system, and they were only proved to be one system by running them as one.
Conclusion
A major incident stops being a Slack thread and a pasted-together doc. The declaration is a row
(bridges, one per bridge, its scope and clock auditable), the lifecycle is a row per transition
(bridge_events), and the scope is a typed graph traversal (cmdb_impact, impact_rels-filtered — not
a blanket hop that over-counts) rolled up to the business services it takes down — all over the same bucket
your ITSM already lives in. The bridge opens on real dependency data, runs open → mitigating → resolved,
and closes the ticket it was declared from with it — one connection, no model, one copy of your rows:
the tickets (SQL) and the CMDB (Graph).
Two of those rows are guarded, and it is worth remembering which. POST /declare pages the org;
POST /bridges/{id}/state stops the SLA clock. Those are the only two actions here whose consequences leave
the database, so those are the only two that carry a permission — while the queries a commander runs every
minute of an outage carry none, because a permission everyone must hold protects nothing. And the one route
in the neighbourhood with no identity at all is the SLA tick, because a machine heartbeat has no user to be.
Gate what escapes; leave the ordinary work ordinary.
Next steps:
- Build a ServiceNow-style ITSM on DataK3 — the masters + the CMDB graph this bridge scopes against.
- CMDB Blast Radius on DataK3 — the typed graph this skill traverses, assembled from first principles.
- Auto-Triage Incidents on DataK3 — how the P1 that gets declared here was graded and routed in the first place.