Testing
Testing selects scenarios where bias is a risk, sets granularity, chooses a metric with reference values, records each test and justifies the method and its interpretation.
What clause 9.2.4.3 asks
The test work has a fixed shape. Pick situations where bias is a risk, following risk-based testing practice. Decide how finely to split the groups. Choose a measure and reference values. Write down each test and what it produced. Write and justify a method for examining results for bias against affected people, against the risk work and against earlier findings. Write and justify the rules for reading those results before the run. Run the system, read the output against the business requirement and record the reading. Treatments may be recommended, but the document does not specify them.
Why it matters
The clause is a protocol, not a metric choice. The scenario decides what is tested, the granularity decides who is visible, the metric decides what is measured, and the interpretation rule decides what the number means. Each choice can hide a problem. A scenario that never occurs, a granularity that merges small groups, a metric that assumes balance, or an interpretation rule written after the result will each produce a confident but empty finding. The reference values are the organization's own thresholds, which is why clause 5.2.5 says the standard sets none.
What good looks like
Write the test protocol before running it. Name the scenario, the group granularity, the metric, the reference value, the interpretation rule and the owner. Record the raw outputs, not only the summary. Interpret against the business requirements and record the interpretation in the evaluator's words. Where a subgroup is too small to test, record that and the alternative method used.
Evidence an assessor asks for
The assessor will ask for the interpretation rule as it stood before the test. A rule that appears only in the final report is not evidence of a pre-committed method.
- Test protocol Scenario, granularity, metric, reference value and interpretation rule, fixed before the run
- Raw outputs Each test and its outputs, retained for re-analysis
- Interpretation record The result read against the business requirements, with the reasoning
- Small group note Any group too small to test, with the alternative method
Where it commonly fails
- The metric is chosen after the result
- Small groups are merged and the merge is not recorded
- The interpretation is a threshold comparison with no business context
What to do before publication
Freeze the protocol before the run. Then the test can produce a finding the team did not expect.
Read across
- 5.2.5 Evaluation in the life cycle
- 9.2.3 Evaluation of design and outputs: outputs
- 9.2.4 Evaluation of design and outputs: actions
- 9.2.4.4 End of the evaluation
- D Measuring Bias
← 9.2.4.2 Examine data pre-processing9.2.4.4 End of the evaluation →