Evaluation of design and outputs: actions
The actions examine the records, the data pre-processing, the testing, the mitigation and optimization effects and the UI and UX design.
What clause 9.2.4 asks
Four groups of tasks make up the evaluation. Study the requirements, the data record and the design record. Study the data preparation for bias within the method itself and for bias the method introduces, using the data part of the profile. Choose test situations where bias is a risk, set the level of detail, choose a measure and reference values, record every test and its result, write and justify the method and the reading rules before the run, test, read the results against the business requirement and record the reading. Then compare human and system decisions where useful, study bias introduced by treatments and by performance tuning, and check the interface and experience design against the attributes of the people identified.
Why it matters
The action structure follows the path from requirements to data to test to interpretation to design. Each group has a different failure mode. A records review can miss a requirement that was never written. A pre-processing review can miss a transformation that encodes a proxy. A test can use a scenario that never occurs in practice. An interpretation can use a rule that was chosen after the result. A UI review can miss a control that excludes a group. The clause asks for all five.
What good looks like
Run the actions as a sequence and keep a record for each. For the pre-processing review, list every transformation and its effect on the protected attributes. For the testing, write the scenario and the interpretation rule before the test. For the end review, compare the UI and UX design with the current stakeholder attributes and record every gap. Then link the findings to the evaluation report.
Evidence an assessor asks for
The assessor will ask whether the interpretation rule was written before the result. A rule written afterwards is a way of making the result acceptable.
- Records review The requirements, data and design documents examined, with the version
- Pre-processing review Each transformation and its effect on the protected attributes
- Test protocol The scenarios, metrics, reference values and interpretation rules, written before the test
- UI and UX review The design compared with the stakeholder attributes, with every gap recorded
Where it commonly fails
- The pre-processing is reviewed only for performance
- The interpretation rule is chosen after the test
- The UI and UX review is a visual check with no stakeholder attribute list
What to do before publication
Fix the scenario, metric and interpretation rule in writing before the test. Then the test can surprise the team.
Read across
- 9.2.1 Evaluation of design and outputs: purpose
- 9.2.3 Evaluation of design and outputs: outputs
- 9.2.4.1 Examine the records
- 9.2.4.2 Examine data pre-processing
- 9.2.4.3 Testing
- 9.2.4.4 End of the evaluation
← 9.2.3 Evaluation of design and outputs: outputs9.2.4.1 Examine the records →