26 Jul

The Costs of Misclassifications

Written by

I started to think about the consequences of medical misdiagnoses. What happens if someone has an infection and is told they do not by their doctor? What if someone does not have an infection but is treated as if they do? Which mistake would be worse?

 

 

Did you know some hospitals use robots to clean hospital rooms to reduce the risk of infection? I read an article online about this the other day and I started to think about the consequences of medical misdiagnoses. What happens if someone has an infection and is told they do not by their doctor? What if someone does not have an infection but is treated as if they do? Which mistake would be worse?

To answer these questions it helps to think in terms of the cost of a mistake and to do this let’s look at two hypothetical classification models and their confusion matrices:

jul26 1

A confusion matrix has a simple interpretation. The one for Model A says 630 people were correctly predicted to not have an infection (True Negatives), 150 people were correctly predicted to have an infection (True Positives), 170 people were incorrectly predicted to not have an infection (False Negatives), and 50 people were incorrectly predicted to have an infection (False Positives).

Model A and B behave differently. Model A favors false negatives (170; lower left corner) whereas Model B favors false positives (200; upper right corner), and Model A’s misclassification rate is lower (22% vs. 27%). Is Model A superior to Model B? In terms of the misclassification rate it is, but misclassification rate does not take the costs of a mistake (false positive or false negative) into account.

Considering Costs

One obvious way to views costs is in a monetary sense. Patients predicted to have an infection are tested by the hospital for infections including the type. Those that test positive for infections are moved to a new room, treated for their infection, and their old room is sanitized with the cost being:

  1. Tests for an infection: $2,000
  2. Sanitizing a room and moving a patient to a new one: $5,000
  3. Treating an infection early: $20,000
  4. Treating an infection later: $30,000

This information defines a cost matrix so that accounts for the severity of false positives and false negatives.

jul26 2

My rationale for this matrix was

  1. If a patient is predicted to not have an infection and truly does not then there is no cost (True Negative)
  2. If a patient is predicted to have an infection and does not then the cost is the $2,000 for the tests (False Positive). Also, for simplicity I assumed the tests run by the hospital were always correct.
  3. If a patient is predicted to not have an infection but does then the hospital spends $2,000 for the tests to determine the type of infection, $30,000 to treat the infection due to the later discovery, and another $5,000 to clean the room and move the patient for a total of $37,000 (False Negative)
  4. If a patient is predicted to have an infection and does, then the hospital spends $2,000 to run the tests to determine the type of infection, $20,000 to treat the infection due to the early discovery, and $5,000 to clean the room and move the patient for a total of $27,000 (True Positive)

Another way to view the costs of an incorrect prediction is in terms of the costs that the hospital will always incur:

  1. If a patient has an infection (i.e. Actual = “Infection”; second row in the cost matrix) then the hospital has to treat them (regardless of what the model predicts) and always spends at least $27,000 ($27,000 if the model correctly predicts a disease and $37,000 if it does not)
  2. Similarly, if a patient does not have an infection (i.e. Actual = “No infection”; first row in the cost matrix) then the hospital does not need to treat them and thus will always spend at least $0 ($0 if the model correctly predicts no infection and $2,000 if incorrectly predicts an infection)

The minimum cost for patients with an infection is $27,000 and $0 for those without, and I subtracted the minimum cost of $27,000 from the values in the bottom row (Actual = “Infection”) and $0 from the values in the top row (Actual= “No Infection; the top row does not change). My new cost matrix, which is equivalent to my first, is:

jul26 3

This matrix has a valuable interpretation. If the model predicts a false negative (lower left corner) it costs $10,000 more than it would otherwise cost to treat a patient whereas if the model predicts a false positive (upper right corner) then it costs $2,000 more than it would otherwise.

Using the Cost Matrix

A cost matrix can be used to evaluate the cost of a model (or with the model’s loss function to build entirely new models that minimize cost instead of purely model error). Model cost is computed by multiplying the number of false positives by $2,000 and the number of false negatives by $10,000. The final cost values for each model represent the cost incurred beyond what it normally costs to treat patients for an infection:

Model A: 170 * $10,000 + 50 * $2,000 = $1,800,000

Model B:   70 * $10,000 + 200*$2,000 = $1,100,000

Model B is superior because its predictions result in a lower total cost even though Model A has a superior misclassification rate. Model A classifies more patients correctly overall, but predicts more costly false negatives.

What about the patients?

Model B results in a lower overall cost, but what about the patient? In this example a false negative is the most expensive option and is also very damaging to patients because allowing infections to go untreated can be fatal or have severe side effects. If a situation arises where minimizing costs have detrimental effects on patient health then the cost matrix should be adjusted to reflect this. For instance, if the treatment for an infection is dangerous, then false positives would be more serious (perhaps even more serious than false negatives) and the cost matrix would have to be updated to reflect this. Numerical values may not be available to account for potential treatment risk to patients, so experimenting with different values in the cost matrix may be necessary.

Different mistakes have different costs and recognizing that a model should incorporate asymmetric costs results in improved performance that incorporates practical considerations. Costs are an important concept in classification models and should always be considered in any analysis. 

Charles Harrison

Charles Harrison is a Marketing Statistician at Salford Systems. In this role he uses his statistical modeling and computer programming background to illustrate the practical techniques and uses of Salford Software in real-world applications in tandem with our in-house marketing team.

Charles is a recent graduate from the University of Central Florida, where he earned his Masters of Science in Statistical Computing with an emphasis in Data Mining. Charles worked as an intern at Sprint Nextel in their Network Planning Division and later was a Business Intelligence Intern, taking his expertise to a new level by creating an infrastructure designed to analyze M2M data using SQL. He later improved upon his own software to map key geospatial traffic components to optimize the Sprint network. This work carried over to The J.M. Smucker Company where Charles was an internal consultant to marketing teams, providing them statistical analysis for time series forecasting, geospatial, and demographic analysis.

With advanced knowledge in machine learning, logistic regression, experimental design, and marketing, Charles provides actionable insights and valuable results to the Salford Team and translates them to an array of educational, promotional, and consulting projects. Outside of the office, Charles enjoys reading history and playing basketball.

Latest from Charles Harrison

Exploring the world of data mining and the triumphs and challenges experienced first hand from our in-house data scientists.

About Us

a blog by Salford Systems Subscribe Here