What is the difference between Test Suites and Scenario Sets?

Test Suites are small collections of hand-written cases (dozens) with explicit expected outputs — used as release gates during authoring. Scenario Sets are DVS-scale bulk runners (tens of thousands of rows) reporting coverage KPIs across a realistic corpus. Use both for a robust promotion workflow.

Can test suite failures block promotion?

Yes. The governance workflow surfaces the most recent suite run result when a reviewer is deciding whether to approve a version. A failed suite appears as a warning; the reviewer can override it with an explicit comment but cannot ignore it.

Does DecisionManager support behavioral equivalence testing?

Yes. The Test Suites page mounts a behavioral equivalence panel: run the same corpus against two ruleset versions and flag every case where the output differs. This is the primary validation step in an ODM migration.

Decision Testing

Two testing instruments. One governed release process.

Test Suites are hand-written release-gate tools. Scenario Sets are DVS-scale bulk runners with coverage KPIs. A robust promotion workflow for regulated rules uses both — at different stages, for different audiences.

Start free trial Book a demo

Test Suites vs Scenario Sets

Dimension	Test Suites	Scenario Sets
Scale	Dozens of hand-written cases	Tens of thousands of rows
Input format	JSON per case in the browser	CSV / JSON bulk import
Result model	Pass/fail per case, field diff	Coverage KPIs, verdict distribution
Audience	Rule author, pre-review	QA / analyst, pre-promote
ODM analogue	Decision Center test suite	Decision Validation Services
Governance gate	Release gate check	Promotion prerequisite

Test Suites — the release gate

Per-case field diff

Every case shows which output fields passed and which diverged — with the expected vs actual value side by side. Not just 'failed': exactly where.

Governance integration

The reviewer's approval screen shows the most recent suite run result. A failed suite is surfaced explicitly; the reviewer cannot ignore it without an override comment.

Behavioral equivalence

Run the same corpus against two versions and flag every case where the output differs. The primary validation step in an ODM migration.

Scenario Sets — DVS-scale coverage

Ingest tens of thousands of rows from CSV, JSON, or a production traffic replay. Run as a batch. Report on coverage KPIs — not just whether cases pass.

Rule coverage

What percentage of rules in the decision table fired at least once across the corpus? Low rule coverage reveals dead logic.

Condition coverage

What percentage of condition partitions were exercised? Catches untested edge conditions at the boundary of numeric ranges.

Verdict distribution

APPROVE / DECLINE / REVIEW / ERROR breakdown across all inputs. Compare distribution between versions to catch unexpected shifts.

Verdict drift threshold

Fail the scenario run if the APPROVE rate drifts by more than N percentage points vs. the baseline corpus run. The threshold is configurable per pipeline.

Recommended promotion workflow

1
Author + hand-write test suite
Rule author
Build 40–60 cases covering every business requirement, edge condition, and known regression. All must pass before submitting for review.
2
Run Scenario Set baseline
QA analyst
Run a 50k-row corpus against the current production version to establish a KPI baseline.
3
Run Scenario Set against new version
QA analyst
Compare verdict distribution vs. baseline. Flag any drift above threshold.
4
Governance review
Reviewer
Reviewer sees suite run result + scenario comparison summary. Approves or requests changes.
5
Pipeline deploy
Automated
Pipeline runs analysis check → suite run → scenario run → deploy to green slot → traffic shift.

Both instruments are included on every plan

Test Suites and Scenario Sets are available in the free trial. No add-on, no separate licence.

Start free trial Book a demo