DecisionManager ships two testing instruments with different purposes. Test Suites are hand-written, per-case, release-gate tools. Scenario Sets are DVS-scale bulk runners with coverage KPIs. Understanding which to use — and when to use both — prevents coverage gaps at the point of promotion.

## Two instruments, different jobs

Decision Manager ships two testing surfaces that look similar but serve different purposes:

Test Suites — the release gate

Test Suites (`/suites`) are the small-suite tool. A suite is a collection of hand-written test cases: each case has a JSON input payload and an expected output. You build them during authoring — one case for the APPROVE path, one for DECLINE, one for each edge condition in your decision table.

The runner sends one HTTP call per case to the Decision Server and folds every response into a single result row. A case passes when the actual response matches the expected payload on the fields you chose to assert. A case fails when any asserted field differs — the diff shows exactly which field diverged and by how much.

**Suite runs are a release gate**: the governance workflow surfaces the most recent suite run result when a reviewer is deciding whether to approve a promotion. A failed suite blocks promotion; the reviewer must either fix the rule or explicitly override with a comment.

Behavioral equivalence (XOM honesty)

The Suite page also mounts the behavioral equivalence residual: given two ruleset versions (A and B), run the same corpus of inputs against both and flag any case where the output differs. This is the Validate step in an ODM-style migration: does version B produce the same decisions as version A for every known input?

Scenario Sets — DVS-scale coverage

Scenario Sets (`/scenarios`) are the bulk runner. You ingest a corpus — tens of thousands of rows from a CSV export, a production traffic replay, or a generated test dataset — and run it as a batch against the engine. The results report on **coverage KPIs**, not pass counts:

- **Rule coverage**: what percentage of rules in the decision table fired at least once across the corpus? - **Condition coverage**: what percentage of condition partitions were exercised? - **Verdict distribution**: APPROVE / DECLINE / REVIEW / ERROR breakdown across all inputs.

This is the ODM Decision Validation Services analogue. The goal is not to verify that every case produces a specific expected output — it's to verify that the ruleset's behaviour across a realistic input population is what you expect at the aggregate level.

Typical workflow for a regulated promotion

A robust promotion workflow for a credit underwriting ruleset uses both:

1. **Test Suites**: 40–60 hand-written cases covering every explicit business requirement, edge case, and known regression. All must pass before the author submits for review. 2. **Scenario Sets**: A 50,000-row corpus (last quarter of production decisions, anonymised). Run against both the current champion and the proposed new version. Compare verdict distributions: if the APPROVE rate changes by more than 0.5pp, flag for human review. 3. **Governance review**: The reviewer sees both suite run results and the scenario set comparison summary before approving.

When to use each

**Use Test Suites when:** - Authoring a new rule or modifying an existing one - You have explicit expected outputs for specific inputs - You are building regression coverage for known-bug fixes - You need a release gate for the governance workflow

**Use Scenario Sets when:** - Promoting a major version change - Validating a migration from IBM ODM (behavioral equivalence at scale) - Checking coverage across a realistic input population - Comparing verdict distributions between champion and challenger

Running from the CLI

Both surfaces share the Live `dm test` headless runner (same suite / scenario-set run APIs as the Manager console). There is no `dm suite` or `dm scenarios` subcommand — those names are residual marketing fiction.

bash
# Run a test suite and exit non-zero on failure (CI gate; JUnit for CI tabs)
dm test --ruleset <id> --suite <suiteId> --version <versionId> \
  --junit artifacts/junit.xml

# Run a scenario set (bulk) against a pinned version dm test --ruleset <id> --scenario-set <setId> --version <versionId> \ --out-dir artifacts/test

# Champion/challenger distribution compare is a console Scenario Sets feature # (coverage / differential UI) — not a separate dm CLI subcommand. ```

Test Suites vs Scenario Sets: Two Testing Instruments for Different Scales