Pursuit of Perfection, 2: Overfitting

One of the reasons why action plans are not fully realized is that some people are afraid of stumbling upon the worst-case scenario when they are doing what they are supposed to do.  They are opening a door (often literally) not knowing what they will be greeted by.  It feels better, or rather, safer to leave the door closed.  Let us examine an example of this, No Pass Zones.

No Pass Zones are a popular technique to address call-lights on inpatient floors.  A patient presses their call-light, and a light goes off in the hallway, so anyone walking by can see that the call-light button was pressed.  The idea is that anyone walking by can see the light on and walk into the room to address the concern.  Any employee, clinical or non-clinical, is responsible for entering the room and identifying the request and starting the resolution.  Now often this plan fails because the person in the hallway is heading off to do some specific task and cannot be distracted.  But, for non-clinicians, the fear is that the patient request will be something that they cannot help with or are incapable of helping with.  What if the patient has some complicated question about medication?  What if the patient needs help going to the bathroom?  What if the patient has fallen out of bed, cracked their head on the floor and is bleeding out?  This fear of the unknown will keep people from entering the room and assessing the situation.¹ When I would present at hospitals, I would often discuss this as people fearing the N^th case.  So, we miss the dozen opportunities to raise/lower the bed, move the patient’s phone closer to them, show them how to operate the TV, or close the blinds, because we are afraid that someone will ask for something that is outside our licensure or comfort zone. 

When presented with this problem, workgroups will build action plans that account for these potential outcomes.  While this is exactly what a workgroup should be doing—addressing significant obstacles and challenges—sometimes, this work gets caught up in the weeds of a problem and tries to create provisos and addendums to address every possible permutation of a problem.  This may make sense in the abstract, but what ends up happening is that they create new problems that they didn’t have before.  In creating a storehouse for all possible issues and opportunities, they create a plan that is so cumbersome that it is difficult to use and may be ignored or misused.  This creates the “policy trap.”  An external surveyor (from CMS, Joint Commission or the state) will evaluate your performance against your own stated policy standards.  The more detailed your policy, the more likely you will be found out-of-compliance with your own rules.  This is why your Quality and Compliance folks will always favor more general language than specific to-dos when assisting with any policy construction.  The rules that the state or the federal government give you are complex enough without compounding the problem by doing it to yourself.  Keep it simple.

Now in many of my essays, I run the risk of painting a scenario that is so extreme that it seems unlikely to occur.  Often my point, though, is to clearly illustrate the issue rather than recounting a specific real-world example.  In this situation, though, it is also to highlight a real problem of structuring solutions to problems.  We want to create a solution that is good at addressing several different eventualities.  But, if we build a solution that is good at addressing ALL the eventualities, we have fallen into the trap of overfitting our solution. 

I came across the concepts of “underfitting” and “overfitting” in my life as a statistician, though they exist in other universes as well.  They are two extremes in efforts to create solutions to data problems.  At one extreme, you create a simple model that might explain some variance but is so simple that it is unclear if it is actually useful in explaining any variance at all.  A million years ago, I was watching Mike & Mike on ESPN one morning and Mike Greenberg was so excited by a stat that he saw that 54% of all NFL playoff teams win their first football game of the season.  He saw this as powerful statistic that would predict the end of the season by the beginning of the season.  This does have the patina of usefulness, until you realize that you could achieve essentially the same accuracy by simply flipping a coin.  This is a clear example of underfitting a model.  It is not wrong so much as it is so simplistic as to only be right by chance. 

Overfitting is a bit harder to explain.  To avoid underfitting, you start adding variables or contingencies to better explain current data.  But at some point, you add so many variables as to lose the real effectiveness of the model.  Probably the easiest way to explain this is with this meme.

You want to build a doghouse.  Obviously, you want to build it wide enough and long enough so the dog could comfortably lie down and tall enough so the dog could comfortably sit up.  But if you push the model further, by adding additional measurements and adjustments, you have created something that is highly accurate—certainly tailored to all aspects and dimensions of the dog, but also completely useless.  This is the great irony here.  In an effort to build a model that takes in as much data as possible, one creates a model that is excellent at explaining a seriously small set of circumstances.  The more input included, the less useful the output.

One reason why people will overfit a model is that they forget what they are trying to accomplish with the model.  Yes, they want to create a model that explains the data that they have, but they forget that what they are REALLY trying to do is create a model that will help with future scenarios.  But the better they are at explaining variance in their current space, the worse they are at being flexible enough to account for future events. 

Using this logic in action planning, the more detailed a solution you create to address potential issues, the less likely it will be useful in addressing the N^th case.  So, imagine that the goal is to implement No Pass Zones in the hospital. 

Underfit model: If you see a call-light illuminated, go in and address the need.  This certainly addresses the key behavior but provides no insights other than telling the person to enter the room.  It will likely be ignored by many people since it does not allay their fears over entering the room.

Overfit model: If you see a call-light illuminated, go in and follow the following steps

• If you are non-clinical and the patient asks about medication, do A
• If you are non-clinical and the patient asks about a test scheduled for today, do B
• If you are non-clinical and the patient asks about a test scheduled for a future day, do C
• If you are non-clinical and the patient asks about a past test, do D
• If you are clinical but not directly related to this patient’s care and the patient asks about medication, do E
• If you are clinical but not directly related to this patient’s care and the patient asks about a test scheduled for today, do F
• If you are clinical but not directly related to this patient’s care and the patient asks about a test scheduled for a future day, do G
• If you are clinical but not directly related to this patient’s care and the patient asks about a past test, do H
• If you are clinical but not directly related to this patient’s care and the patient asks about a test scheduled for today, do F
• If you are clinical and you are directly related to this patient’s care and the patient asks about a test scheduled for a future day, do H
• If you are clinical but not directly related to this patient’s care and the patient asks about a past test, do I
• Et cetera, et cetera, ad nauseum…

In attempting to address all possible scenarios, it becomes unwieldy and unhelpful.  It does not help people do the task you ask and the rule’s own complexity will prevent people from wanting to act on the broad principle of trying to help patients and staff.

Properly fit model: If you see a call-light illuminated, go in and assess the situation

• If the request is clinical and urgent, do A
• If the request is clinical and not urgent, do B
• If the request is non-clinical and urgent, do C
• If the request is non-clinical and not urgent, do D

This model helps people know how to assess the situation and act, rather than simply trying to create a beast of a decision-tree that will think for them.  You will have to train people, especially non-clinicians, on what is clinical versus non-clinical and urgent versus not urgent.  It might even mean that you end up creating a category for “semi-urgent” or some other differentiation.  But you have both created a model that most people can follow, and you have created a model that is easy to troubleshoot.  If the process fails, you have an easy way to evaluate the situation.  Perhaps the person thought that helping a patient move from the bed to the chair was non-clinical/urgent or considered the question about the scheduled PT visit was clinical/non-urgent.  In the end, though, it is easier to focus making the underlying concepts clear rather than trying to account for every possible scenario.

Overfitting is an example of trying to make the perfect model rather than a good useful model.  Much like the last essay on not letting the perfect be the enemy of the good, establishing the balance here between under- and overfitting is a judgement call.  My general rule-of-thumb is that a process should account for a significant majority of cases and should address a significant majority of concerns or expectations.  But it should also be flexible enough to handle different unexpected situations by creating some broad standards that accept that people are sentient creatures trying to provide the best outcome for any situation.  If you assume people are idiots who don’t have your patients’ best interests at heart so they need to be constrained to address all situations appropriately, action-planning may not be for you.  Instead, you might have a conversation with HR about why they are hiring such troglodytes in the first place.²

¹What is often not appreciated is how NOT entering actually can make the situation worse.  Personally, if I had fallen and was bleeding out on the floor, I would prefer to be found quickly by a food service person, rather than waiting for a nurse to eventually find me.

²I did have a paragraph here comparing hospital action-planning to being the dungeon master in a game of Dungeons & Dragons, but I cut it for clarity and length. I put it here as a nod to my fellow travelers.