--- name: decube-incident-remediation description: >- Use this when someone asks how to fix, resolve, remediate or act on a data-quality incident or a failing monitor such as but not limited to null check failures, duplicates, stale table, volume drop, format errors — or asks what the next steps are on an incident. For adding new monitoring coverage to an asset, use the decube-monitor-recommendation skill instead; once an incident is remediated, it is a reasonable follow-on. --- # Skill: Decube Incident Remediation The Incidents page says *what broke*. This skill says *what to do about it*. Take one incident — or one failing monitor — and produce a remediation plan: the diagnosis, the likely root cause, who is affected downstream, the exact next steps, and how the fix gets verified and the incident closed. This is a **read-heavy, advise-then-act** workflow. Everything up to the plan is read-only. Trusty does not run SQL and does not change data: it hands the user the queries and the steps. The only writes are on the incident itself (`assign` / `mute` / `close` / `reopen`), a monitor recalibration, a monitor enable/disable, and an optional monitor re-run — each only after the user says yes. **This skill runs in two environments, and they don't expose the same tools.** On the Decube **MCP** the incident write tool `update_incident` is available; **inside Trusty it is not yet** (the read tools all are). Never assume a write tool exists — **check whether it is actually in your available toolset before you offer the action it performs.** When `update_incident` is absent (Trusty today), the diagnosis, root cause, impact and next steps are unchanged and just as valuable — you simply deliver the plan and hand the close-out to the Decube web app instead of doing it. Treat that as the normal Trusty path, not a degraded one. This same "is the tool present?" check applies to any action tool the skill names; `update_incident` is just the one known to differ between the two environments today. ## When to use this skill - "Incident #412 — how do I fix this?" / "What are my next steps here?" - "The null check on `orders.customer_id` failed. Why, and what do I do?" - "`payments` is stale / row count dropped / duplicates appeared — walk me through fixing it." - A user is looking at an open incident and wants a plan, not a restatement of the alert. **Scope is one incident (or one failing monitor), in → plan, out.** If the user asks "which incidents should I work on first?" across the org, that is a *prioritisation* question, not this skill — answer it with `list_incidents` triage (severity, age, unassigned), then offer to run this skill on the one they pick. If they name several incidents, handle them one at a time; don't blend two root-cause stories into one plan. ## Tools this skill uses | Step | Tool | Type | |---|---|---| | Find the incident | `list_incidents` | Read | | Read the incident + timeline | `get_incident_details`, `get_incident_history` | Read | | Read the rule that fired | `get_monitor_details`, `get_monitor_history` | Read | | Resolve the affected column/table | `get_asset`, `search_assets` | Read | | Check for other failing checks on the same asset | `get_quality_summary` | Read | | Actual data distribution | `get_profiling_list` → `get_profiling_result` | Read | | Upstream root cause / downstream impact | `get_lineage`, `list_incidents` per neighbour | Read | | Find the person to hand it to | `find_user_by_id` (the incident's own owners), `find_user_by_email`, `find_users_by_name` | Read | | Re-run the monitor to verify | `run_monitor` → `get_monitor_run_status` | **Action** | | Recalibrate a miscalibrated rule | `update_monitor` | **Write** | | Pause / retire a noisy scheduled monitor | `enable_disable_monitor` | **Write** | | Assign / mute / close / reopen | `update_incident` | **Write — MCP only; not in Trusty yet.** Check it's present before offering; otherwise hand off to the web app. | ## Workflow 1. **Resolve the incident — and read everything it already hands you.** If given an id, go straight to `get_incident_details`. If given an asset or a monitor name, use `list_incidents` (scoped with `asset_id` + `asset_type`, `status` defaults to `open`), confirm which incident you're working, then **still call `get_incident_details`** — it is the only call that returns `monitor_id`, which step 2 needs. From `get_incident_details`, note: `monitor_id`, the affected `asset_id` / `asset_type`, the `incident_type` and `test_type`, the `mode`, the `incident_level`, `first_seen` and `last_seen`, the current `status`, the `assignee`, and the `description` — which usually states the breach outright ("Expected upper bound to be 0.00. Actual value instead is 3.0."). Also note **`data_owners` and `business_owners`**: these are the people the fix gets handed to, and they resolve with `find_user_by_id`. You should not be guessing at an owner or asking the user who owns the table. `find_user_by_id` returns that person's name and **email** — which is exactly the `assignee_email` `update_incident` needs, so the whole assign chain resolves without the user supplying anything. **Read `can_edit` before you plan any close-out — and note it comes only from `list_incidents`.** `get_incident_details` does not return it. So on the id-entry path, take the `asset_id` + `asset_type` you just read and make one scoped `list_incidents(asset_id=…, asset_type=…, status=None)` call to find this incident's `can_edit`. If it is `false`, the connected user cannot assign, mute or close the incident — offering to do so produces a rejected write. Give the plan anyway, and name who does have access instead. If you can't establish `can_edit` at all, say the close-out is unconfirmed rather than asserting permission in either direction. `get_incident_history` can legitimately return an empty `histories` list — don't build the timeline on it. `get_monitor_history` (step 2) is the reliable run series. 2. **Read the rule that fired.** `get_monitor_details` with the `monitor_id` from step 1 — the `test_type`, `mode`, `threshold_type` (`auto` vs manual), `threshold_min` / `threshold_max`, `dimension`, and for a Custom SQL monitor the `custom_sql` itself. Then `get_monitor_history` for the run series: what value breached, by how much, and whether this monitor has been stable until now or has been flapping. `test_type` and `mode` are **already in the step 1 payload** — you can start the diagnosis fork before this call returns, and you should know `mode` long before step 7. What `get_monitor_details` adds that you cannot get earlier is the **threshold configuration**, and that is what decides whether "widen the bound" is even a possible remediation: an `auto` monitor has no manual bounds to widen (see step 3). 3. **Diagnose: is the data wrong, or is the rule wrong?** Apply the fork below. This is the step that makes the answer advice instead of a summary — do not skip to remediation. State the verdict in one line and the evidence for it. 4. **Find the root cause upstream.** A column check fires on a `property`. `get_lineage` accepts `asset_type: "property"`, and Decube's column-level lineage is the more precise signal — **try the property first**, and fall back to the parent `dataset` (resolve it with `get_asset`) when column-level coverage is thin. On each meaningful upstream asset (`dataset`, `data_job`), run `list_incidents` — a correlated incident, a **schema-drift** hit, or a **job-failure** hit upstream is usually the actual cause, and the incident in front of you is a symptom. Say so explicitly when you find one: fixing the symptom is the wrong move. **Pass `status=None` on these upstream lookups.** `list_incidents` defaults to `status="open"`, and the upstream job failure that caused today's nulls has very often already been closed — the default silently hides the most likely cause. Widen `lookback_days` past its 30-day default too when `first_seen` is older than that. For a `custom_sql` incident the affected asset is the **monitor**, not a table or column, so there is no parent dataset to walk from — resolve the tables named in the monitor's SQL with `search_assets` and run lineage from those. 5. **Size the blast radius downstream.** `get_lineage(direction="downstream")` — again from the property where column-level lineage exists, otherwise the parent dataset. List the dashboards, reports, charts and jobs that consume this data — this is the "who do I need to warn" half of the plan, and it is the part the Incidents page cannot tell them. Flag anything with an owner or a business-critical name. `get_quality_summary` on the asset is worth a call here too: other failing checks on the same table often turn one incident into a coherent story rather than an isolated alert. 6. **Produce the plan.** Use the output shape at the bottom: diagnosis, root cause, impact, the ranked next steps, and the verification step. Every step must be *actionable by a named party* — "ask the `orders` pipeline owner to re-run yesterday's load", not "investigate the pipeline". Resolve that named party from step 1's `data_owners` / `business_owners` rather than describing them generically. Include the advisory SQL from the playbook so the user can localise the bad rows themselves. 7. **Verify — but check the monitor's `mode` first.** How the fix is confirmed depends on the monitor, and only one kind can be re-run on demand: - `mode = on_demand` → offer `run_monitor` (a write-ish action: ask first), then poll `get_monitor_run_status` with the returned `job_id`. Only one on-demand run can be active per monitor at a time. - `mode = scheduled` → **`run_monitor` is not available.** Say plainly that the next scheduled run is the verification, and that close-out should wait for it rather than happening now. Do not propose a re-run you know will be rejected. - **Schema-drift and job-failure monitors can never be re-run**, whatever their mode — they're system-detected. Verification there is the upstream fix landing. 8. **Close out on the incident — if you can.** Two gates, both must pass before you offer a transition, and they fail to the same place: - **Is `update_incident` in your toolset?** On the MCP, yes. In Trusty today, no — so don't offer assign/mute/close; deliver the plan and point the user to the incident's status controls in the Decube web app (right panel of the incident). - **Is `can_edit` `true`** (from step 1)? If not, the user lacks edit access on the asset; name who does instead of offering a write that will be rejected. When both pass, offer the transition that fits the diagnosis — assign, mute, or close — under the rules in "Close-out actions" below. Never close an incident whose fix hasn't actually been verified; recommend assign-or-mute instead and say why. Everything *before* this step is identical in both environments — the plan is the product; the close-out is a convenience when the tool is there. ## Diagnosis: is the data wrong, or is the rule wrong? Both are real remediations. Fixing data when the rule is miscalibrated just re-fires the incident next run; recalibrating a rule when the data genuinely broke hides a live problem. Discriminate on the signals you already read in steps 1 and 2: | Signal pattern | Verdict | Remediation direction | |---|---|---| | `auto` threshold, long stable history, **first breach**, sharp step change | **The data changed.** | Fix upstream / backfill. Leave the monitor alone. | | Breach coincides with an upstream job failure or schema-drift incident | **The data changed, upstream.** | Remediate the upstream cause; this incident is a symptom. | | Manual threshold, tight bound, monitor **flaps** in and out repeatedly | **The rule is miscalibrated.** | Widen the bound with `update_monitor`. (Moving to `auto` needs an already-`scheduled` monitor.) | | **`auto` threshold**, monitor flapping or firing on small deviations | **The model is too sensitive.** | **Sensitivity feedback, not `update_monitor`** — there are no manual bounds to widen. See the next section. | | Breach is small, within normal seasonality visible in `get_monitor_history` | **The rule is too tight** for this asset's natural variation. | On `auto`: lower sensitivity via incident feedback. On a manual threshold: widen the bound. | | Business rule genuinely changed (new optional field, intentional backfill, deprecated column) | **The rule is now wrong.** | Update or retire the monitor; document the change. | | Breach recurs on a predictable cadence (weekend loads, month-end) | **Expected condition.** | Mute for the window, or re-scope the monitor. Note scheduled Freshness already models quiet periods — a weekend breach *there* points at sensitivity, not muting. | **Before you propose any recalibration, check `threshold_type`.** The single most common way to give useless advice here is to tell a user to widen a bound on an `auto` monitor, which has none. `auto` → model feedback. Manual → `update_monitor`. Custom SQL → always manual. Corroborate a "data changed" verdict with the profiler where it helps — `get_profiling_list` → latest completed run → `get_profiling_result` — to see the current distribution against what the rule expects. Remember profiler percentages are decimals (`0.2` = 20%). If the profiling run predates the breach, say so rather than reasoning from stale stats. If the evidence is genuinely ambiguous, say which single check would settle it (usually the "when did it start" query below) rather than guessing a verdict. ## Remediation playbooks by test type The SQL below is **advisory** — Trusty has no query tool and runs nothing. Emit it for the user to run in their own warehouse, and resolve the asset's parent `source` first so the dialect and the fully-qualified name are right. Substitute real table, column and timestamp names; never hand back a template with `` left in it. ### `null` — completeness broke The highest-value query is **when the nulls started**, because it dates the breach to a specific load and that points at the pipeline change: ```sql SELECT DATE(load_ts) AS load_day, COUNT(*) AS rows_loaded, SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_rows FROM analytics.orders WHERE load_ts >= CURRENT_DATE - INTERVAL '30' DAY GROUP BY 1 ORDER BY 1 DESC; ``` Then inspect the offending rows to see what they have in common (one source system, one region, one job run): ```sql SELECT * FROM analytics.orders WHERE customer_id IS NULL ORDER BY load_ts DESC LIMIT 100; ``` Next steps, in order: identify the load that introduced them → fix the transformation or the upstream source → backfill the affected partition → verify per step 7. A `NOT NULL` constraint or a `COALESCE` default at the ingest layer is the durable fix; say so, but note it's a schema change the owner has to approve. ### `unique` — duplicates appeared ```sql SELECT order_id, COUNT(*) AS copies FROM analytics.orders GROUP BY order_id HAVING COUNT(*) > 1 ORDER BY copies DESC LIMIT 100; ``` Almost always a **double load** or a broken merge/upsert key. Check upstream `data_job` runs for a retried or duplicated execution before assuming the data is genuinely duplicated. Remediation: de-duplicate the affected partition, then fix the merge key or the job's idempotency — deleting rows without fixing the load means it happens again. ### `freshness` — data stopped arriving ```sql SELECT MAX(load_ts) AS last_load, CURRENT_TIMESTAMP AS now_ts FROM analytics.orders; ``` This is a **pipeline** problem far more often than a data problem, so go to lineage first: look for a failed or skipped upstream `data_job`, an upstream freshness incident, or a schedule change. Remediation is re-running the load, not touching the table. Two things to know before calling a freshness monitor wrong. Scheduled Freshness uses an **ML model trained on historical arrival patterns** — it fires when the probability of arrival drops below 50%, and it already accounts for expected quiet periods like weekends and off-hours. So "it fired on a Sunday" is not automatically a miscalibration. And if the monitor is watching the **wrong timestamp column**, that is a genuine rule fix — but see the retrain warning below, because changing it clears the monitor's history. ### `volume` — row count off ```sql SELECT DATE(load_ts) AS load_day, COUNT(*) AS rows_loaded FROM analytics.orders WHERE load_ts >= CURRENT_DATE - INTERVAL '30' DAY GROUP BY 1 ORDER BY 1 DESC; ``` A **drop** means a partial or missing load — find the gap, re-run it. A **spike** means a duplicate load — check for a double execution, then de-duplicate. Volume baselines against this table's own pattern, so a breach is a genuine deviation rather than a threshold someone set badly. If it's over-firing on a table with naturally lumpy loads, that's a sensitivity question (Volume supports Smart Training), not a bad bound. ### `email` / `uuid` / `regex_match` — format/validity broke ```sql SELECT email, COUNT(*) AS occurrences FROM analytics.customers WHERE email IS NOT NULL AND email NOT LIKE '%_@_%.__%' GROUP BY email ORDER BY occurrences DESC LIMIT 100; ``` (Use the warehouse's own regex function where available — `REGEXP_LIKE`, `RLIKE`, `~` — and for `regex_match` use the monitor's configured `value` pattern.) Look for a single bad source, a test/placeholder value, or an encoding change. Remediation: fix at the point of entry and clean the existing rows. If the pattern itself is too strict for legitimate values, that's a rule fix, not a data fix. ### `avg` / `min` / `max` / `string_length` — range broke ```sql SELECT MIN(amount), MAX(amount), AVG(amount), COUNT(*) FROM analytics.payments WHERE load_ts >= CURRENT_DATE - INTERVAL '7' DAY; ``` Check for unit changes (cents vs dollars), sentinel values (`-1`, `9999`), and currency or locale shifts — these masquerade as data-quality failures but are usually contract changes. A genuine business shift means the threshold needs recalibrating. ### `cardinality` — distinct values drifted ```sql SELECT status, COUNT(*) AS rows_with_status FROM analytics.orders GROUP BY status ORDER BY rows_with_status DESC; ``` A **new** category usually means an upstream enum was extended without telling anyone; a **missing** one means a source stopped sending it. Both are contract conversations with the upstream owner, and both often warrant a glossary/description update so the next person isn't surprised. Cardinality tracks distinct counts on a **5-period rolling window** and is scheduled-only, so it needs no training period — a breach reflects the last few scans, not a long baseline. ### `custom_sql` — a business rule was breached Different in kind: the *user* wrote the assertion, so the monitor's own SQL **is** the diagnosis. Read `custom_sql` from `get_monitor_details` and reason from what it asserts, then hand back that query plus a drill-down that lists the offending rows rather than the aggregate the monitor computes. Two structural facts matter here. Custom SQL monitors **do not support Smart Training** — every threshold is manual — so the "rule is wrong" branch is always an `update_monitor` threshold change or a `custom_sql` rewrite, never model feedback. And a `custom_sql` incident attaches to the **`monitor` asset**, not to a table or column, so there is no parent dataset for lineage; resolve the tables named in the SQL with `search_assets` first. ### `schema_drift` — table structure changed Covers `table_addition`, `table_deletion`, `column_addition`, `column_deletion` and `column_type_change`. **System-detected and auto-enabled when a source is connected — it cannot be re-run, updated or disabled through Trusty.** This is rarely a data fix; it's a **contract** conversation: 1. Confirm whether the change was intentional — `get_incident_details` names what changed. 2. `get_lineage(direction="downstream")`. A type change or a dropped column breaks consumers silently, which makes this the highest-value blast-radius call in the whole skill. 3. Warn the downstream owners *before* their queries fail, and get the upstream owner to confirm whether the change is permanent. 4. If it is permanent, the follow-on work is updating dependent models — and a dropped or renamed column may have orphaned monitors that now need re-creating on the new column. Verification is not a monitor re-run; it's the downstream assets being updated. Only close once that has actually happened. ### `job_failure` — an ETL job failed Auto-created when an ETL-type source (dbt, Airflow and others) is connected. **Also system-managed: no re-run, no update, no disable through Trusty.** The incident is on the job, but the damage is downstream of it, and that's the part worth surfacing: 1. `get_incident_details` for which run failed and when. 2. `get_lineage(direction="downstream")` from the `data_job` — every dataset it writes is now stale or partial, whether or not those tables have fired their own incidents yet. 3. Check those datasets with `list_incidents` (**`status=None`**) — where they are already firing freshness or volume incidents, name them as symptoms of this one rather than separate problems. 4. The fix is in the user's orchestrator. Trusty can identify the job, the failed run and the owner; it cannot retry it. Verification is the next successful run of that job. ### A test type not listed above The `test_type` enum is wider than these playbooks — `range`, `value_is`, `value_in`, `is_past_date`, `is_future_date`, `zero_percentage`, `negative_percentage`, `sum` and others exist. **Don't silently improvise and present it as a playbook.** Fall back to the general shape: read what the monitor asserts and what value breached from `get_monitor_details` + `get_monitor_history`, write the advisory query that reproduces *that* assertion against the real table, run the diagnosis fork as normal — and say plainly that you're reasoning from the monitor's definition rather than a pre-written playbook. ## When the verdict is "too sensitive" on an `auto` monitor: model feedback **An `auto` (Smart Training) monitor has no `threshold_min` / `threshold_max` to widen, so `update_monitor` is the wrong tool.** The product's mechanism for this is **incident model feedback**: thumbs-down on the incident reveals a **sensitivity slider** from −5 to +5, which sets the model's confidence interval. | Slider | Confidence interval | Effect | |---|---|---| | −5 | 0.99 | Widest — least sensitive, fewest alerts | | 0 (default) | 0.90 | Balanced | | +5 | 0.80 | Narrowest — most sensitive, most alerts | Each step moves the interval by 0.02, and the new sensitivity takes effect on the next scan. - Available for **Freshness, Volume and Field Health monitors with `auto` thresholds**. - **Not** available for Custom SQL monitors or monitors on manual thresholds — those recalibrate through `update_monitor`. - **This is not exposed as a Trusty tool.** Don't offer to do it. Tell the user exactly where it is — open the incident → 👎 → sensitivity slider — and which direction to move it and why. - On a monitor less than a few weeks old, say so instead of retuning. Early scans don't yet reflect the data's true variance, and the ML model needs **5 valid data points in the last 30 observations** before it raises incidents at all. Too little history is a reason to wait, not to adjust. This is the loop closing — the incident teaches the detection model. Say that out loud: it's the difference between silencing an alert and improving the monitor. ## When the verdict is "the rule is wrong": recalibrating with `update_monitor` `update_monitor` is a **write** — propose the specific parameter change, get approval, then apply it. Only pass the fields you're changing. Check these gates *before* proposing, not after a rejection: - **`test_type` and the monitored asset are fixed at creation.** If the right fix is a different test or a different column, the answer is "delete and re-create this monitor", not an update. Say that plainly. - **There is no delete tool in Trusty.** "Delete and re-create" is a **Decube web app** action — say where it happens rather than implying you can do it. The Trusty-side lever for pausing or retiring a noisy monitor is `enable_disable_monitor`, which **only applies to scheduled monitors** (on-demand monitors have no enabled/disabled state to toggle). - **A config change can wipe the monitor's training history — check before you propose.** Changing the scan frequency, the row-filtering mode, or the **timestamp column** (`timestamp_column_id`) triggers a fresh retrain, and the monitor's previous history is **cleared** as part of it. That destroys the very baseline this skill's diagnosis reasons from, and it affects open incidents on that monitor. Disclose it before proposing any of these, and note or resolve open incidents you want to preserve first. A wrong `timestamp_column_id` is a real and common freshness miscalibration — but fixing it costs the history, and the user should choose that knowingly. - **`threshold_type`, `threshold_min` and `threshold_max` only take effect together.** To replace a threshold, pass `threshold_type` with the new bounds; omit it to keep the current one. `threshold_min` / `threshold_max` are **integers**. - **`auto` thresholds require a `scheduled` monitor — and `update_monitor` cannot change `mode`.** So "move it to `auto`" is only a valid proposal for a monitor already running `scheduled` (check `mode` from step 1). For an `on_demand` monitor, `auto` means delete-and-re-create as scheduled in the web app; offer a wider manual bound instead if they'd rather not lose the monitor's history. - **Schema-drift and job-failure monitors are system-managed** — they can't be updated here at all. Point the user at the Decube web app. - Other useful levers for a miscalibrated rule: `lookback_period` (widen the window for a noisy signal), `incident_level` (downgrade a warning-worthy check firing as critical), and `value` (loosen an over-strict `regex_match` pattern). Always state what the change gives up. Widening a bound or downgrading severity reduces noise *and* sensitivity — the user should be choosing that trade knowingly, not just making an alert stop. ## Close-out actions on the incident These actions all run through `update_incident`, **which exists on the MCP but not in Trusty yet.** So before this section applies at all: **Gate 1 — is `update_incident` available in this session?** If it isn't (Trusty today), none of the transitions below are possible from the skill. Don't describe them as if they are. Deliver the plan and tell the user where to act: open the incident in the Decube web app and use the status controls in the right panel (assign via the **Assignee** field; close / mute / reopen via the status controls). This is the expected Trusty flow — say it plainly, not apologetically. **Gate 2 — is `can_edit` `true`?** Only relevant once gate 1 passes. It comes from `list_incidents`, not `get_incident_details`, so resolve it as described in step 1. If it is `false`, none of the transitions will succeed even with the tool present — the connected user lacks edit access on the underlying asset. Deliver the plan, say plainly you can't action the incident, and name the owner who can. When both gates pass: `update_incident` is a **write** — every one of these needs explicit confirmation of that specific action, and is attributed to the connected user in Decube's audit trail. | Diagnosis | Action | Rules to respect | |---|---|---| | Someone else owns the fix | `assign` | Requires `assignee_email`. Resolve the person from the incident's own `data_owners` / `business_owners` with `find_user_by_id`, or by name/email with `find_users_by_name` / `find_user_by_email` — **never ask the user for a raw user id.** | | Known/expected condition, or waiting on an upstream fix | `mute` | Requires `mute_duration` — `1D`, `1W`, or `1M` from now. **Only valid on an open incident.** Pick the duration from when the condition actually clears. | | Fix verified, monitor passing | `close` | Valid on open or muted incidents. Don't close on a *claimed* fix — only a verified one. | | Closed too early, or it came back | `reopen` | Only valid on a closed or muted incident. | **Trusty has no comment action** — there is no way for this skill to log the remediation plan onto the incident. Don't imply otherwise and don't offer it. (The web app's incident audit history does record actions, and thumbs-up/down model feedback exists in the UI, but neither is exposed as a tool here.) Deliver the plan in the conversation, and if it needs to live somewhere durable, the honest options are assigning it to the owner so it lands in their queue, or putting the context in the asset's description via the catalog. Say which you're doing. ## Keep this in sync with the docs **The live docs are the source of truth; this file is a cache.** Before recommending a close-out transition or a monitor change — and always if a tool call is rejected or the rules here look stale — reconcile with `search_docs` / `get_docs_page`: - Incident statuses, closing and muting: `https://docs.decube.io/data-quality/incident-management` - What the incident detail view shows: `https://docs.decube.io/data-quality/incident-details` - Monitor types, thresholds and modes: `https://docs.decube.io/data-quality/available-monitor-types` - Sensitivity and model feedback: `https://docs.decube.io/data-quality/incident-model-feedback` - Retraining, training thresholds and monitor behaviour: `https://docs.decube.io/data-quality/known-limitations` If the live docs contradict this file, **follow the docs**, and tell the user plainly that this skill looks out of date — naming the specific rule that changed — so it can be fixed. One exception: where the docs describe something the **web app** can do, that is not evidence Trusty can do it. Tool capability is set by the tool schemas, not the docs. ## Boundaries — be honest about these - **Trusty does not run SQL, change data, or re-run pipelines.** It reads Decube metadata and hands back queries and steps. Never phrase advisory SQL as something that was executed, and never report a data fix as done. - **Trusty cannot fix an upstream job.** It can identify the job, the owner and the run that broke — the fix happens in the user's orchestrator. - **The monitor itself can be changed** (`update_monitor`) when the verdict is "the rule is wrong" — but only within the gates above: not the `test_type`, not the asset, not the `mode`, never a system-managed monitor, and never without disclosing a retrain that clears history. - **Trusty cannot delete a monitor and cannot set sensitivity.** Both are web-app actions. Name them precisely and point the user there rather than offering to do them. - **Incident close-out isn't always available.** `update_incident` is on the MCP but not in Trusty yet — check it's in your toolset before offering assign/mute/close, and hand off to the web app's incident status controls when it isn't. Even where the tool exists, Trusty can't act on an incident the user can't edit: check `can_edit` (from `list_incidents`, so on the id path you need the extra scoped call from step 1) before offering. - **If the evidence doesn't support a verdict, say so.** A stale profiling run, no lineage coverage upstream, or a monitor with too little history are all reasons to name the gap instead of asserting a root cause. A confident wrong diagnosis costs more than an honest "here's what I'd check next." ## Output shape Lead with the verdict — the user already knows something is broken. The names below are **illustrative only**: every owner you name must come from the incident's `data_owners` / `business_owners` resolved through `find_user_by_id`, never from inference. If you can't resolve an owner, write the role ("the `orders` pipeline owner") rather than inventing a person. > **Incident #412 — `Not-null — orders.customer_id`, critical, open since 14 Mar 09:00** > > **Diagnosis: the data changed.** Auto threshold, 90 days stable, first breach — 4.2% nulls > against an expected upper bound of 0. This is not a miscalibrated rule. > > **Likely root cause:** upstream job `dbt_orders_staging` has a job-failure incident from > the same run window (already closed, which is why it isn't on your open list). The nulls > almost certainly arrive from its partial output — **fix that first; this incident is a > symptom.** > > **Downstream impact (3 consumers):** `Revenue Daily` dashboard, `Finance Weekly` report, > `customer_ltv` job. Worth warning Priya Raman (owner, `Finance Weekly`) before they read > today's numbers. > > **Next steps** > | # | Step | Owner | > |---|---|---| > | 1 | Run the "when did the nulls start" query below to pin the load date | You | > | 2 | Fix and re-run `dbt_orders_staging` for the affected partition | Alex Tan (data owner) | > | 3 | Backfill `orders` for that partition | Alex Tan | > | 4 | Add a `NOT NULL` guard at ingest so this can't recur | Alex Tan (schema change) | > > **Verification:** this monitor is `scheduled`, so it can't be re-run on demand — the next > scheduled run confirms the fix. Keep the incident open until it passes. > > **Want me to assign this to Alex Tan?** (That closing offer assumes `update_incident` is available and `can_edit` is `true`. In Trusty, where the tool isn't present, end instead with: *"To assign this, open the incident in Decube and set the Assignee to Alex Tan."*) Then the advisory SQL, labelled as something for the user to run. Close by offering the one incident action that fits the diagnosis — a single concrete offer, not a menu of four — but only when the `update_incident` tool is present **and** `can_edit` allows it. When either gate fails, point to the web app's incident status controls and name who can act instead.