Data map
The map connects data attributes to the impacted stakeholders, is repeated after retraining, and records every imbalance.
What clause 7.4.2.3 asks
The data is matched to the affected people and their attributes. The match must allow groups to be compared, including where a group is small or represented unevenly. Three tasks follow: match the data to the attributes from the stakeholder step, do it again after any retraining, and write down every imbalance. The data is then watched as the evaluation clause requires, and the profile is updated with what is found.
Why it matters
The map is the test of the representation claim. A team can say the data is representative; the map shows which group is represented, at what size and with what gaps. The retraining requirement is the one most often missed. A model is retrained on new data, the old map is left in place, and the evaluation is run against a population that no longer matches the data. The continuous monitoring link closes the loop with clause 9.
What good looks like
Map each stakeholder group and attribute to the data, and record the count, the share and the known gaps. Record small samples explicitly, because a small group can be invisible in an aggregate metric. After every retraining, create a new map version and link it to the model version. Then connect the map to the monitoring plan so a change in representation is a trigger for review.
Evidence an assessor asks for
The assessor will pick a model version and ask for the map that was current when it was evaluated. A map that has been overwritten cannot answer.
- Map table Stakeholder group, attribute, count, share and gap
- Small sample note Every group below the agreed threshold, with the effect on the analysis
- Retraining link The map version tied to each model version
- Monitoring link The map fields that are monitored under clause 9.3
Where it commonly fails
- The map is created once and never repeated after retraining
- Small groups are merged into an other category
- The map is not connected to the monitoring plan
What to do before publication
Version the map with the model. Make the retraining link a required field in the model release record.
Read across
- 6.3 Outputs
- 7.1 Purpose
- 7.4.2.1 Collection and creation conditions
- 9.2.2 Evaluation of design and outputs: inputs
- 9.3.4.2 Each iteration of the ongoing evaluation