One of the challenges in getting people to buy into a process is convincing them that the effort is worth the outcome. As a result, a lot of statistical work I would do was proof-of-concept work. I would pull data, break it into buckets of those that had an intervention and those that did not and see if this created a change in the data. Invariably, while the intervention would improve scores, there would often be one hospital in a system, or one unit in a hospital where the improvement was much less noticeable, or, even, one where there would be no difference. Sometimes, even, the scores would be worse with the intervention than without the intervention. Speaking to an audience that might be resistant to change, that one red flower in a sea of yellow flowers might make people hesitant to try something new. In other cases, if the process was already installed, sometimes that odd-man-out might cause compliance to erode. As the key to successful PX work is in building the consistency and muscle-memory to continually execute on behaviors, a lack of unanimity in these results can sidetrack conversation, slow roll-out, or cause compliance to backslide. So, why does data behave, at times, in messy ways? How do you talk about patterns when they are not always black and white? In this essay I will talk about some things to keep in mind when using data to demonstrate effectiveness.
Before diving deeply into this topic, I must call out an often-unspoken expectation with data. It is easier to implement and defend a process that is so obviously effective that it needs no explanation. Often audiences want a smoking gun. They want epistemic truth that if they do X, then Y will absolutely occur. I understand this thinking, even if it makes me roll my eyes. Everything we do to change ourselves is hard. Whether it is practicing the piano, or exercising, or cleaning the house, or learning calculus, we do it because we want the benefits of that change even if we dread the work to realize that benefit. So, we want to make sure that the strategy we use to improve is likely to actually improve us. But sometimes that quest for the perfect guaranteed success strategy delays our start. We don’t try something until we are confident it will work. In healthcare this can be compounded when your audience contains clinicians and administrators that expect a clear line between cause and effect, usually to a degree that their own discipline cannot meet.
It would be easy to kick a doctor in the shins for expecting a clear cause-and-effect relationship in patient experience when they also accept that sometimes people who get flu vaccines still end up getting the flu. Or when the Marketing department pushes back on slow unsteady improvement when they celebrate that their advertising campaign improved hospital usage by the same slow unsteady rate. In truth, most of us expect clear lines between cause and effect in areas where we are not expert. Heck, my writing over one-hundred essays on patient experience and math is testament to the fact that I believe that there is a cause-and-effect to best-practice PX processes, even if those practices don’t manifest earth-shattering improvements overnight or are immediately obvious.
The reality is that most things are only obvious in hindsight. Moving forward, there is very little that is obviously beneficial, that is also obvious in how to execute. We all know we should eat right and exercise, but there are industries built upon what those two things really mean. We will always need teachers to separate the useful from the extraneous. We need help standardizing behaviors and consistently executing them. We need help learning, especially about those things that we don’t know that we don’t know.
All of this essentially means that there is never a smoking gun, a thing that is so obvious and essential in explaining something that it is impossible to get distracted by anything else. Or, in the language popularized by Nate Silver, it is NEVER all signal and no noise. In the real world, there are confounding variables, and nonconforming datapoints. There are units with high variability and those resistant to change. Everything a statistician tells you is an interpretation of the data. Ideally based upon good math, solid logic, and common sense. But only a snake-oil salesman would try to convince you that all of your patient experience problems can be solved with one simple hack that the big corporations don’t want you to know. To that end, here are a few things to keep in mind when the data isn’t perfect.
Different Execution
Unless you are a single small critical access hospital, it is going to be difficult to build a identical process surrounding any objective. As hospitals become part of larger and more diverse systems, the common processes they adopt (like leader-rounding on patients, post-discharge phone calls, collaborative rounding, etc.) will morph to meet the culture, staffing, and needs of the constituent hospitals, clinics, or units. Sometimes these variants will not get in the way of execution and outcome. Sometimes they will.
For example, I worked with a multi-hospital system in Wisconsin that was returning to leader rounding with patients at the bedside. On their survey, they had a question asking if the patient recalled a leader round during their stay. So, after the reintroduction of the process, it was easy to break the data out into two groups—those that recalled the round and those that did not recall a round—and compare the scores they gave.1 At every hospital, those who recalled a round were giving higher scores on communication and, more importantly to the system, on the Overall Rating question than those that did not recall a leader round. The differences were anywhere from 8% to 15% higher for those who did versus those who did not recall a round. OK, almost every hospital. There was one hospital where the scores were virtually identical and one hospital where the scores were actually lower with patients who recalled a round.
It was easy to spot the reason why the one hospital did not show an improvement based upon rounding. It was because their scores were already high. That is, the baseline was already exceeding goal, so the leader round was not providing a significant bump on top of that already high performance. The other was a bit more puzzling until we realized that this hospital focused their leader rounds on patients who had made a complaint or required some service recovery. Since they were targeting patients who were upset (and not all patients), the reported comparison was not useful. Given how they targeted their rounds, their data simply showed that happy patients gave higher scores than unhappy patients. Stop the presses. If the same intervention does not report the same impact, the first question to ask is whether the intervention is really the same.
A system in North Carolina had hospitals all doing their own thing with PX initiatives, so they set out to standardize their processes. When they targeted universal performance on post-discharge phone calls, they ignored the hospitals already doing them and focused training on the hospitals that were not already doing them. Again, they had a question on their survey asking if the patient recalled a post-discharge phone call, so showing proof-of-concept was straightforward. The data showed that the hospitals with new post-discharge call processes got a nice bump in their scores, but the hospitals who were already doing them did not have any significant bump. It turned out that the hospitals were not all defining a post-discharge phone call the same way. The things that the old-guard were calling post-discharge calls were focused on clinical concerns. They included questions like:
- Did you get your prescriptions filled?
- Did you have a follow-up appointment with your primary care provider?
- Did you have any questions or concerns about your care?
Those who got the roll-out training included those clinical questions, but also asked:
- How do you think we did?
- Is there anyone at the hospital you would like to recognize?
- Is there something we could have done better?
So, while everyone labeled the process the same, the actual execution was much different. While asking the clinical follow-up questions is important, without asking more expansive and patient-centered questions, they missed an opportunity to really connect on the experience.
Different Audience
I have written before on the impact of patient demographics, like age, gender, race, socio-economic status, education, and so on, have on patient perceptions. Likewise, much is written on the different experiences that a surgical patient, a medical patient, and an OB patient are going to have, especially as related to expectations. So, an examination of the effectiveness of any process needs to consider the predispositions that the audience might have. For example, many hospitals make use of various QR codes to access food service options, available TV channels, or even logging compliments and complaints. While this can certainly be convenient, reduce clutter in the room, and minimize infection prevention concerns, it can divide a patient population between those who own and are comfortable with smart phones and tablets and those who don’t and aren’t. So new moms may give higher marks to “Ease of Access” questions than elderly medical patients do when confronted with the same service.
I saw this with a hospital system that implemented a new automated phone tree system for routing clinic appointments and conversations. It was a net positive for many of the larger clinics, especially those with both primary and specialty care, allowing for more seamless connections to the right people. But it was a dissatisfier with many of the patients of the smaller primary care clinics. For the complex clinics, making one call with three keypad prompts was faster than calling four different numbers. For the smaller clinics, patients found the phone tree more complicated and impersonal than just calling the clinic and talking to Denise, like they had done for the past twenty-five years. Expecting the same thing to resonate in the same way with different audiences is unreasonable.
Complex Systems are Complex
Finally, I return to an undercurrent for a lot of my math essays. It is unreasonable to expect that a single simple analysis is going to capture all the nuance in a dataset. I know that my brothers and sisters in the Church of Nerd were probably already wondering why I was talking about simple bivariate comparisons when I should have been doing multivariate logistic regression to truly understand the impact of leader rounding, post-discharge phone calls, or any other element discussed here.2 Beyond that, though, there are a host of confounding variables, or sample biases, or externalities that can influence why patterns seen in many of the areas are not visible in ALL of the other areas. Expecting unanimity in all patterns means drastically reducing the number interesting observations you can make.
As an independent consultant, I found it was easy to confess ignorance on some situations where the pattern is not universal. I was not from there and would never imagine that I know more than the folks working the hallways every day. I found that, not only does this confessed ignorance allow me to avoid wandering onto thin ice, but it also actually gives me an opportunity to invite discussion. A vast majority of the interesting revelations I have discovered in working with clients start with me finding something that bucks a trend that I cannot explain. I then can throw this out to the group and moderate the conversation instead of having all the answers. People like being part of the problem-solving process, so letting them theorize is much better than pulling ideas out of the air. Plus, when they come up with an answer, they are far more likely to own the solution as well. If they look at the data and say, “The reason that pattern doesn’t hold at Clinic X is because they just lost a longstanding doctor without warning” then they can talk about how to address that problem. Even if you work in the health system, you are not required to have every answer. Being able to say, “I noticed that pattern as well and I am not sure why that is. What do you think?” puts you on the offense by saying that it is not YOUR problem to solve, but OUR problem to solve.
1I will quickly point out that this was simply a recollection of a round, which for my purposes was better than a log-book indicating a round. The goal was that memorable rounds led to higher scores, so I cared less about whether they got a round and more on whether they remembered that they got a round.
2To be clear, I do use a number of different tools to evaluate the patterns in the data, but I usually present the simplest version that captures the pattern. It is easier to present an answer that is understandable, if incomplete, than to present the right answer that forces everyone’s eyes to glaze over.
Leave a comment