Webhooks & Event-Driven Decision Automation: At-Least-Once Delivery with Idempotent Receivers

## Events from the governance lifecycle

DecisionManager's Admin console ships a webhook engine that fires events on governance state transitions and execution milestones. The event model covers:

| Event type | Fires when | |:--|:--| | `ruleset.published` | A ruleset version transitions to `published` state | | `ruleset.approved` | A reviewer signs off on a version | | `ruleset.rejected` | A reviewer rejects a promotion request | | `deployment.created` | A version is deployed to a Decision Server environment | | `deployment.rolled_back` | A blue/green rollback completes | | `experiment.promoted` | A challenger is promoted to champion | | `execution.budget_exceeded` | An execution hits the step budget limit | | `warehouse.lag_critical` | The warehouse writer lag exceeds threshold |

Delivery guarantee: at-least-once

The webhook dispatcher uses an at-least-once delivery model with exponential-backoff retry. On a retryable failure (5xx, connection timeout), the dispatcher retries up to 10 times over 24 hours with backoff starting at 30 seconds. On a non-retryable failure (4xx other than 429), the dispatcher marks the delivery as permanently failed and does not retry.

**Implication**: your receiver must be idempotent. The same event may be delivered more than once, especially on intermittent network failures.

X-DM-Delivery: the deduplication header

Every delivery carries the `X-DM-Delivery` header — a stable UUID for that (event, endpoint) pair. The same event re-delivery always carries the same `X-DM-Delivery` value. Your receiver should store processed delivery IDs and skip re-processing if the ID is already in the store.

python
@app.route("/dm-webhook", methods=["POST"])
def handle_dm_event():
    delivery_id = request.headers.get("X-DM-Delivery")
    if is_already_processed(delivery_id):
        return "", 200   # idempotent — already handled
    
    event = request.json
    process_event(event)
    mark_processed(delivery_id)
    return "", 200

Delivery diagnosis AI panel

The Admin console's webhook page mounts a delivery diagnosis panel. For any endpoint experiencing repeated failures, the panel:

1. Fetches the last N delivery logs (response code, response body, latency) 2. Surfaces the failure pattern: is it a consistent 4xx? Intermittent 5xx? Timeout? 3. Generates a diagnosis with suggested fixes, informed by the response body content

This is not a magic box — the diagnosis is generated by the AI authoring layer and always surfaces the raw evidence (response codes, bodies) alongside its interpretation.

Receiver implementation patterns

Governance-triggered CI

json
{
  "event": "ruleset.published",
  "workspace": "ws_abc",
  "ruleset_id": "loan-eligibility",
  "version_id": "v12",
  "published_by": "[email protected]",
  "published_at": "2026-08-05T09:12:00Z"
}

On `ruleset.published`, trigger a CI pipeline that runs integration tests against the newly-deployed version and posts the result back to the team's Slack channel.

Execution anomaly alerting

On `execution.budget_exceeded`, post to a PagerDuty incident — budget exceeded signals that a rule is consuming far more steps than expected, which may indicate a logic error or an adversarial input.

Rollback notification

On `deployment.rolled_back`, post to the release channel with the rollback reason (pulled from the event payload) so the team knows which environment was affected and who triggered it.