decision systems are now used to rank applicants, set prices and allocate inspections.
Their appeal is : the same input receives the same output at four in the morning.
Consistency, however, is not fairness, and a consistent system can be consistently wrong.
enters through the training data, which records past decisions rather than correct ones.
A hiring model trained on a decade of hires learns the preferences of the people who made them.
Removing the protected does not remove the problem, because other fields carry the same information.
A postcode ethnicity in most cities, and a career gap parenthood.
Fairness itself has several definitions that cannot all hold at once.
Equal error rates across groups and equal treatment of equally qualified individuals are mathematically when base rates differ.
Choosing between them is a political decision wearing technical clothing.
is the least glamorous and most effective remedy available.
Recording what the system was trained on, what it is for and what it must not be used for prevents quiet reuse.
Most damage in deployed systems comes from a model applied to a population it was never built for.
A right of appeal to a named human closes the loop.
It is also the requirement that organisations resist most, because it makes the volume of decisions visible.