A treatment can appear better in every and worse in the combined data, without any error having been made.
This is Simpson's , and it is not a so much as a warning about how groups are formed.
The mechanism is simple: the s differ in size and in risk, and the combination weights them accordingly.
A surgeon who takes the hardest cases will have worse overall figures than one who takes the easiest.
She may nevertheless be better within every category of difficulty, and frequently is.
Published that do not adjust for therefore reward selection rather than skill.
Surgeons understand this perfectly well and respond to it, which is the more serious consequence.
A system that punishes bad outcomes without adjusting for risk produces refusals rather than improvements.
The patients declined do not appear in anybody's figures, which is why the effect is hard to detect.
is possible and it introduces a second difficulty that is less often discussed.
Adjusting requires a model of what makes a case difficult, and that model is built from past data.
If past practice systematically disadvantaged a group, the model will treat that disadvantage as an expected outcome.
A hospital performing badly for that group will then appear to be performing as expected.
Risk adjustment can therefore conceal the very that publication was intended to expose.
There is no arrangement that avoids both problems, and the choice between them is a choice about what to make visible.
The defensible practice is to publish both figures and to state what each of them cannot show.
Readers handle two numbers with a better than one number with a footnote.
What they handle worst is discovering later that a single published figure was a choice.