Validation: What Makes Measures Useful?

I have written in the past about how work in patient experience is misunderstood because its presence on a hospital pillar assumes an equivalence with other pillar items like quality, personnel, or finance that doesn’t exist.  This isn’t meant to cast shade by implying that quality or finance are easier to improve.  They are certainly not easy.  It is simply stating that by giving PX equal footing to other hospital goals—which is definitely a good thing—PX is required to create measures demonstrating improvement, just like all the other hospital goals.  We cannot be pleased that our work is getting appropriate attention and then turn around and say, “Trust me, bro” when it comes to proof of performance.  Saying that it is difficult to pick appropriate measures is not an excuse for ignoring or discounting the measures selected.  This essay then will revisit some of the challenges that make PX more complicated than other pillar measures to track and trend and then discuss ways to solve that complication.

Measurement Goal

Forgive a pedantic opening, but it bears stating that all measures, pillar or not, have one primary goal and that is to help move the organization towards a better position relative to some strategic objective.  Finance measures are meant to prepare an organization for growth (or post-COVID, the more modest goal of stopping the bleeding).  Quality measures are designed to provide better outcomes and satisfy state and federal rules on providing best-practice care.  Staffing measures are designed to reduce turnover and improve staff competence.  Patient experience is designed to…  what?  The messaging around PX has the patina of obviousness but is rarely clearly articulated.  In my years, I have heard CEOs and senior executives posit various benefits, all revolving around a broad ill-defined pursuit of “loyalty.”¹

Let us just assume that this is a worthwhile goal.  The question becomes, then, “what is a useful metric to track this goal?”  There are essentially two ways to measure a goal. 

• The obvious solution is to pick a direct measure that explicitly speaks to the goal.  If a hospital wants to measure their financial stability by how they manage their profit margins, then they can track their profit margins.  If a hospital wants to measure quality by its ability to stay in the good graces of the state and federal authorities, then they can count the number and severity of citations received from reviewers and surveyors. 
• While these measures may speak directly to the pillar, the timeliness of data collection often makes them useless for monthly or quarterly tracking.  This is why organizations will often identify key performance indicators (KPIs) or leading indicators.  These are not directly reflective of the pillar goal, but the assumption is that if these numbers improve, then the desired goals will likely be achieved.

So, for example, hospitals will identify a host of measures that speak to their quality.  It is not uncommon for a Quality pillar measure to include a wide array of inpatient, outpatient, and emergency department measures.  Some are direct counts (like slips and falls), some are performance percentages (like the percentage of stroke patients getting a CT scan within twenty minutes), and some are standardized measures that compare a hospital’s performance relative to expected performance controlling for patient-specific nuances (like length-of-stay compared against expected length-of-stay for similar patients with similar diagnoses and similar comorbidities.) 

There is always a tension between these two types of measures.  Some prefer the one overall measure because it is simple and straightforward, even if the path between individual staff performance and the overall measure is circuitous and possesses confounding variables.  Some prefer a suite of measures because they are more directly tied to staff performance, even if the sheer number of measures makes success feel like an exhausting game of whack-a-mole. 

In patient experience, this is exemplified by those who want to treat Overall Rating or Likelihood to Recommend as the single top-line measure of success, and those who would prefer to create a series of measures, like Nurse Compassion, Doctor Explain Care, Discharge Instructions, and Overall Teamwork, etc.  Now, to be clear, in patient experience, this is not a competition, any more than the nail is in competition with the hammer.  In my thirty-plus years in this space, I have NEVER seen an organization select of suite of measures, all preferring instead to adopt one number to track.  The only debate I have seen is whether Overall Rating or Likelihood to Recommend is the better measure.  At most, an organization may simply select some other measure as a leading indicator.²

Now, as I write this, I fully accept that most people working in patient experience don’t have the power to dictate what measures an organization tracks.  My recommendation is not that you spend time fighting that fight.  But it is important to understand that through-line.  I have certainly seen people get fired because they could not successfully discuss the work that they were doing and the progress they had made, because they could not make the top-line measure move in the way leadership demanded.  My suggestion is that, when called on to explain overall performance, you couch that conversation in terms of work you are pursuing.  As in, “our primary focus this year has been on improving nurse compassion scores, and we have seen this score climb in five of seven units at the hospital.”³

Impact of Samples

Every hospital has big units and smaller units.  Every hospital system has bigger hospitals and smaller hospitals.  If all data reported in the pillar is grouped under one umbrella, it can be easy to forget the constituent parts and in forgetting this, senior leaders miss the unspoken message they are sending to all parties.  For example, a system that groups its data from six critical access hospitals (CAHs) and two large hospitals into one number is essentially telling the CAHs that their data does not matter.  With about three-fourths of the data coming from the two big hospitals, how those hospitals go, so goes the system. 

Now most systems will understand this but fail to appreciate the subtext.  Not only will resources flow to the big hospitals at the expense of the smaller ones, but system plans will be driven by the needs and resources afforded those larger hospitals.  When I worked in the Quality department of a hospital system, it was clear that some of the smaller hospitals struggled with maintaining appropriate insulin levels in diabetic inpatients which was leading to some adverse drug events.  The system could not understand how the flagship hospital performed so well on this, given their volumes and challenges, but the CAHs could not.  After about seven minutes of work (literally, as I read the policy and made one phone call), I realized that the policy assumed a pharmacist was staffed on-site 24 hours a day, seven days a week but there was only ONE hospital in the system that met that policy—the one that was performing well with the measure.  It was that lack of pharmacological consultation that was the gap in performance and that staffing gap was driven by system fiat.  The point of the story is that, even if the system acknowledges differences, if they don’t understand the ramifications of those differences, they are likely to group data in a way that mask confounding variables. 

Even within a hospital, this can be in play.  That hospital that was succeeding in managing insulin was struggling at patient experience.  Their scores fluctuated monthly, much to the consternation of their leadership.  It turned out, though, that when one looked at the data for individual units, the scores were flat.  It was consistent month to month.  The reason why the hospital’s scores fluctuated was not by variance in performance, but by the volume of surveys they got back.  When certain units got a lot of surveys back, their good scores buoyed the overall scores, and when they didn’t, the hospital scores sunk.  By misunderstanding the data, the leaders were chasing problems that didn’t exist and raising the anxiety in their departmental leaders. 

Again, most PX people don’t have the power to change the methodology that a vendor uses for sample selection any more than they can demand a different roll-up methodology for the system pillar score.  Again, my recommendation is that, even if the data is reported as one lump sum, you should do the leg-work to bring some granularity to the data, by reporting it out by hospital or unit.  Even if not all units are large enough to support it, your biggest units can support breakouts.  This will help leadership to refine their message when it comes to who is succeeding, who is failing, and who is hiding in the shadows, since different hospitals or units have different directors or workflows.  It makes sense, then, to not throw out Director A’s baby because of Director B’s bathwater. 

Quantitative versus Qualitative

I have often said that for patient experience, it is more about HOW something is done than whether something is done at all.  This is why when it comes to experience, I am far more interested in qualitative measures (like was as an experience Excellent, Very Good, Good, Very Poor, etc.) than quantitative measures (like did something happen Always, Usually or Sometimes).  However, when it comes to metrics and pillar performance, a reliance on qualitative measures can lead to misunderstanding.  I have spent a lot of time trying to explain why an 8 is not good enough as we chase 9s and 10s, or why a “Very Good” is not sufficient as we chase “Excellent.” 

As a result, while I think qualitative measures are more accurate of an experience, I also think that both the bedside and the boardroom can wrap their heads around quantitative measures more easily.  This means that PX people need to talk about quantitative measures as a steppingstone to useful qualitative targets.  For example,

• It does no good to round on nurses and ask them if they were nice or compassionate today.  Try that and see what response it generates.  Instead, it is more valuable to ask, “what percentage of hourly rounds did you perform?” or “what was the median call-light response time?” 
• Likewise, doctors will roll their eyes, if you ask them, “did you spend enough time with your patients today?”  Instead, ask a question like, “how often did you sit down, when talking to your patients?” or “did you introduce yourself when you entered the room?”

While none of these concrete behavior questions speak to niceness or compassion or explanations or time-spent, the expectation is that if we do these things consistently, we are more likely to be perceived as nice or thoughtful, either because continuous performance of the rote behavior will actually morph into compassion, or, because we are keying upon something that patients have historically identified as important to their qualitative perception.  Since asking staff to self-identify (or asking leaders to measure) qualities like niceness or compassion is fraught with measurement error, this is the best we have. 

This exists in a space where most PX people can own and manage.  This is not trying to change senior leaders’ minds on what is useful information.  In fact, while frontline managers dislike talking about what it means to be nice or compassionate, they often LOVE measures like this.  It takes the guesswork out of what “excellent” looks like and aligns PX work with a lot of the other work they are tasked with.  So, “did you round every hour?” gets added to “did you put a fall-mat down?” or “did you show how to operate the call light?” as tactics that can be easily measured.  Linking this back to the broad qualitative measures on the pillar still requires work, but these measures are an easy shorthand for explaining what work is currently being done.

It is easy to blame senior leaders for being lazy, but that is unfair.  They don’t have time to consider every facet to every problem.    Heck, they may not keep up with how changes in their organization affect pillar metrics.  When I started working with one hospital system, they had eight hospitals.  When I stopped working with them, the system was over twenty hospitals strong.  That growth was incremental, though, which meant that the system woke up one day and realized that some assumptions that were made five or seven years earlier no longer applied.  Leaders’ general desire for metrics that are straightforward and easy to digest lead to measures that can become simplistic or subject to unnoticed undercurrents.  Changes in federal policy, insurance payment schedules, or, say, a worldwide pandemic, can lead to shifts in expectations.  If these are not managed by changes in measures, this dissonance can lead to frustration.  Most PX managers cannot dictate these changes, but they can reflect upon the changes in underlying architecture and ask thoughtful questions to those in charge.

¹Like so many things I reference, the concept of loyalty in healthcare deserves a longer discussion, with I will save for another time.

²And don’t get me started on the fact that most frontline leaders (and therefore frontline staff), HATE KPIs or leading indicators.  Since they only get carrots and sticks based upon the pillar measure, they only care about the pillar measure.

³Like so many other topics, there is a larger essay here on how to parse data to help tell a story.  When I make statements like this, I do document them as future essay topics, but if there is one of my past promises that I have not yet delivered on, feel free to call it out in the comments and I will move that topic up in the batting order.