Aristotle Score criticism

A group of American surgeons involved in the congenital heart surgery databases is working on a scoring system called Aristotle Score.

Aristotle Score

In order to evaluate the surgical results, the 30-day mortality was calculated and used to compare hospitals. But the mortality alone wasn’t a good parameter, because patients are very different from each other. It’s no wonder that a hospital has high mortality, when it treats very sick patients.

To help in evaluating the results, the term complexity was introduced. In order to express the complexity, the Aristotle Score was invented. It is a scoring system, where a numerical value is assigned to each surgical procedure. The more complex procedure, the higher score. At the first glance, it looks simple.

My doubts

Patient to procedure link

Each patient carries a binary response variable with values of dead or alive. How to calculate the mortality vs Basic Score? You can calculate the mortality of a set of patients, not of a set of procedures, because it’s patients who die, not the procedures. How to relate patients to procedures?

One patient can have multiple procedures (see the data structure). Which procedure should have the outcome (dead/alive) assigned to it? When a patient has one procedure, it’s obvious. But what about the second half of patients who had more than one procedure?

The problem was “solved” by discarding procedures and leaving only one procedure per patient. It is equivalent to saying: “Only one procedure influences the outcome. Always.”

Mortality vs Basic Score

When you want to know the ABS for a patient who has two procedures, how do you calculate it? Add both score values? No. You have to discard all the procedures but one.

About half of the patients in the database have more than one procedure. The calculations of ABS are inevitably biased.

Averaging?

You can’t add ABS values of procedures. But existing reports employ averaging of the ABS values (or in other words, calculate the mean ABS)

Above image illustrates the averaging. Please notice the plus (+) signs on the right side. They denote addition. If you’re not allowed to add Aristotle Score values, how come you’re allowed to average them?

The unit?

The unit is not defined. How to interpret the arbitrary numbers that just came out from doctors’ heads? Why are values from 1.5 to 15? Is 6 twice as complex as 3?

Interactions

When two factors are combined, a new effect can be created. Interactions are very important for every medical study.

The idea of expressing the complexity of a patient treatment with a single number means that the Aristotle Score complexity-adjusted analysis method misses any possible interactions between procedures.

How is ABS related to the mortality?

A further effort is currently made to check if the mortality follows the Aristotle Basic Score. Being curious myself, I made a binomial regression with a logit link function. As a result, I got a formula that describes the statistical relation of the ABS to the mortality and can be used to predict mortality from the ABS values. Actually, the regression itself is a process of finding such weights for the parameters, that the prediction precision is maximized.

Having the formula with weights ready, I predicted the mortality for each procedure and compared them to the actual mortality. The actual mortality for each procedure can differ even as much as five times from what’s predicted from the ABC model.

Why not just take the mortality per procedure?

Why invent some arbitrary values and then verify (whatever it means) that they reflect the mortality? Why not just calculate the mortality per procedure? Isn’t it simpler? Advantages:

  • Estimated values are probabilities, real numbers between 0 and 1.
  • You don’t have to check if they follow the mortality, because they are the mortality.
  • You are able to calculate the expected mortality for each surgeon or each center. I already implemented it and sneaked it into the restricted part of the EACTS Congenital Database reports. The expected mortality is one of the options to choose. If you happen to be the EACTS Congenital Database’s member, try it.

Author: automatthias

You won't believe what a skeptic I am.