Evaluation in the life cycle
Evaluation is resourced across the life cycle and begins with how the system uses data and whether it is a solution in its context of use.
What clause 5.2.5 asks
Evaluation also needs resources across the life cycle. It starts by looking at the parts of the system, at how they use data, and at whether the system makes sense as a solution in its setting. People with knowledge of bias in such systems can help. The document allows the team to set thresholds but does not set one; the measurement guidance sits in an informative annex.
Why it matters
The evaluation is often reduced to a test suite run at the end. The clause asks for something wider: an examination of the components, the data use and the fit between the system and its context. That is why competent people are named. A test suite can show a disparity; it cannot decide whether the system is a sensible solution for the people it affects. The absence of a fixed threshold is also important. The organization has to set its own threshold and justify it.
What good looks like
Start the evaluation with a component and data-use review, before the formal tests. Record how each component uses data and where a difference can enter. Then examine the fit between the system and its context with someone who knows the domain. Set the thresholds with a rationale and record who approved them. Resource the evaluation at each iteration, not only at go-live.
Evidence an assessor asks for
The assessor will ask who set the threshold and on what basis. A threshold copied from a vendor default is not a decision the organization can defend.
- Component review Each component, how it uses data and where a difference can enter
- Context fit note The examination of how the system fits its setting as a solution
- Competence record The people who did the evaluation and the bias knowledge they brought
- Threshold rationale The thresholds used, who set them and why
Where it commonly fails
- The evaluation is a single test run at the end
- No one examines whether the system is a sensible solution in context
- The threshold is a default with no rationale
What to do before publication
Set the threshold with the business owner and the affected-group representative before the tests are run.
Read across
- 5.2.4 Risk and impact assessment in the life cycle
- 9.2.1 Evaluation of design and outputs: purpose
- 9.2.3 Evaluation of design and outputs: outputs
- 9.2.4.3 Testing
- D Measuring Bias
← 5.2.4 Risk and impact assessment in the life cycle6.1 Purpose →