Once you establish yourself as someone who can keep the data trains running on time and as someone who can do interesting secondary analysis, people will come to you with other questions or ideas. Or you will come up with good questions as you commute to work or even when you wake up in the middle of the night. Of course, many times these ideas will crash on the rocks, as you realize that you don’t have the right data in the right format to do the work.1 There is no crime in having great ideas that cannot be realized with the current data. You can just move on to the next great idea.
And then there are the zombie ideas; ideas that pop up as new or exciting, only for you to realize that they are not workable and the idea fades away. Then months or a year later they reemerge because you forgot they were unworkable, or because an enthusiastic leader read an article or had a moment of inspiration and ‘gifted’ this work to you. Usually, this excitement bumps up against the reality of the problem, and it dies again. But, on occasion, a change in perspective or data availability does allow for this research to be executed. Not only is that a great day, but it fuels hope that all of the other great unworkable ideas will be resolved someday. One that I have worked on off-and-on over the past thirty years and I have found it to be the king of the zombies. If you have spent any time in the patient experience or customer service realm, you have thought about it as well.
There has always been a desire to link up data sources from patient interviews, employee interviews and even medical staff interviews or marketing research and speak of causal relationships between them. The assumption is that we should be able to connect these data points and determine how movement on one measure impacts movement on other measures. Since it is coveted to so many leadership audiences, I have taken to calling it The Golden Egg.
Anyone who has worked in patient experience long enough has looked for or been asked for this Golden Egg since if we can quantify how these things work together, we can create a strategy that encompasses a number of different workgroups and aligns effort with a single focus. It would validate previously-held beliefs that employee satisfaction begets patient satisfaction and that patient satisfaction can turn community satisfaction. While I will never say never, I have concluded that it is extremely unlikely that anyone will ever find this golden egg. The analysis done in this field fails to be useful because even after it makes massive assumptions in the data, it still finds very little worth talking about. In the next essay, I will discuss the logical issues that make this work challenging. In this essay, I will explore the data issues that make this work difficult.
Before I continue, I will make two quick points.
- There are some who will say that they have cracked this problem, though, they usually point to broad comparisons of top-box responses across different studies. I have never found this to be satisfying for reasons I will talk about below. This work requires a lot of data assumptions, complex data architecture, and ends up with a lot of correlations that are not significant.
- Just because I say that I will never crack this problem, I am not saying that no one will ever crack it. There have been too many times in recorded history where someone will say that something can never be accomplished only to have it become reality, often just a few short years later.2
The data issues are daunting, but not necessarily obvious. In fact, as one glances at the data, there appear some easy paths to success. But as one starts digging deeper, it becomes clear that there are some large issues—including aligning units of analysis, conflicting timeframes, and the amount of needed data—that make this work difficult. Before I begin here, I will insert my usual caveat that I will strive for math-talk that is clear but not pedantic, but if I fail, feel free to call me out in the comments.
Unit of analysis
The unit of analysis is the level at which the data is collected. There are several ways to think about this, but I usually default to a simple rule. If you put your data into a spreadsheet, what does each line of data represent? That is your unit of analysis. If the spreadsheet has 400 lines of data, each representing one patient’s survey responses, then the unit of analysis is an individual patient and their responses. If each line of your data is the amount of profit or loss for the hospital by year, then yearly profit/loss is your unit of analysis.
Knowing your unit of analysis is critical. Since this work (like a lot of work) will require you to connect different datasets, you need to have a common unit of analysis to connect them. For example, imagine you wanted to compare patient HCAHPS scores to hospital profitability and you have two datasets. The first is the hundreds of surveys collected from patients and the second is a list of profit or loss for the hospital by year. How do you connect these two datasets? You cannot combine the datasets as they are, because the units of analysis (individual patient responses vs a year’s profit or loss) do not match-up. Your best strategy will be to summarize the patient data. If you group the scores by year, Instead of having hundreds of individual patient responses, you summarize the scores by year. Once you have converted your unit of analysis from individual patient to YEAR, you can now connect the two data sources. So, for 2024, the PX score was X1 and the profit was Y1. For 2025, the PX score was X2 and the profit was Y2, and so on. From here you can graph it or run a correlation or any other test.
So, if you have a spreadsheet of 1,000 patient responses and another spreadsheet of 1,000 employee responses, how do you combine them. It may seem easy, since they are both individual-level responses, but they are not the same individuals. Consider for a moment how you would link a single patient’s responses to a single staff member’s responses. Unless Joe Snipp is BOTH a patient who took the patient survey AND an employee who took the employee survey, one cannot connect these lines of data. You will have to summarize the data with common units of analysis. This is doable, like our patient scores against profitability, but it requires work and creates its own issues, discussed below.
Conflicting Timeframes
Let us imagine using the same approach with the patient/employee data that we did with the patient/financial data. One could bucket the scores by year. So, we would summarize the data by having Year 1 patient score and employee score, Year 2 patient score and employee score, etc. Problem solved!
Or is it? The challenge with these two datasets is that they don’t have a timeframe that aligns. The two studies use two different research designs. Patient data is collected using an on-going model. Every day, patients arrive and leave hospitals and clinics. So, every day outreach is conducted to capture the patient perception. Your vendor posts that data to the portal as it gets collected. That data is continually updated every day until the end of time, or you change vendors. Employee data on the other hand is collected as a point-in-time survey. Every 18- to 24-months, an organization decides to survey their employees. The survey is constructed and outreach begins. After the window closes (usually in three weeks), the survey is shut down and the results are tabulated. This data will be finalized and will not change.
If you want to summarize the data as referenced above, which Year 1 patient and employee data compared, and Year 2 patient and employee data compared, you will run into issues. First, do you really want to compare the patient data collected between January 1st and December 31st with the employee data collected between August 5th and August 26th? It seems that in this case, one is injecting a lot of error into the comparison, since life in the hospital may have been much different in April as it was in August. Indeed, since you are likely always working to improve the patient’s experience, you would actually HOPE that it was different (and better) in August versus April. Perhaps, then, you only want to compare the patient data collected between August 5th and August 26th. OK, but now you are discarding almost 95% of your patient data. Not only are you losing a lot of precision, your issues with degrees of freedom magnify as well.
Second, given that the window for employee surveys is only every year-and-a-half to two-years, you will not even have data for every year. With the financial example, between 2021 and 2025, you will have five yearly comparisons. But in this model, you are lucky if you get three and there is a high probability you will only have two points of comparison.
Third, even if you are comfortable with the very few points of comparison, that number drops to zero, if you want to factor in nursing or physician opinions as a third point of comparison, since the cadence for those surveys often do not align with the broader employee survey.3
Sheer size of needed data
Let us assume that you decide to compare only the patient data collected during the employee survey window and you are comfortable with only three data points in a five-year period. What will you do with this data? Let us make the math simple by assuming that you run an employee study every year. You have a patient and an employee score for every year. How many years of data would you need to run a correlation? Thirty data points would be serviceable, meaning that you must have data spanning back to 1996, ideally, using the same question and the same scale every year, to run a simple correlation. So, the five years of data you have looks thin. You could run a Pearson’s r on it, but you would likely suffer withering stares or worse when you present that number.
Ah, but you say, we can break the data out more precisely. Instead of looking at overall hospital scores, we can look at the data by discharge unit. So, instead of one data point for every year, we will have a data point for every discharge unit. Excellent! Except that you run into one of two other problems.
On one side, you run into the problem of small sample sizes. When you compare the patient scores for one unit against the employee scores for that one unit, you quickly end up with very small sample sizes. Sure, some units will have forty patient surveys and thirty employee surveys. But for each of those, you will have two or three that have only fifteen patient surveys and only five employee surveys. Summarizing those data points gets difficult because the percentages get very broad. Does it really make sense to include the 33% patient score based off three surveys against the 75% employee score based upon four surveys?
So, you decide to go through the data and only include the bigger discharge units for this comparison rather than all the discharge units. You are now throwing even more data away and not improving your overall n-size at all. For example, imagine that you decide to only keep data from the four largest discharge units in this analysis. You accept that you are throwing away even more patient data and now also throwing away some employee data as well, but it is in the service of a great cause, since we won’t have to wait for thirty years to look at the correlation. Great! The problem is that you have shrunk your window needed to acquire useful data down from thirty years to only eight or nine years. Better, but it still feels like seven lifetimes. The amount of data that you need, just to summarize it honestly, becomes astronomical. Given the reality of survey response rates for both studies, it may not only be astronomical, it also may be impossible to achieve.
All these problems have solutions, of course. You can define your units of analysis along a standard timeframe and seek to maximize your total data points by focusing on a useful level of granularity that balances between too-few data points and too-small sample sizes. Like many problems in life, you can throw money at this, by massively increasing your outreach to get as many surveys as possible. But like many problems in life, there is an upward boundary on how much data you can get and a finite amount of blood and treasure you can expend to harvest this data.
Next time, though, we will discuss the elements that REALLY make this pursuit difficult. While we can bull through the data issues with a big checkbook and an indominable will, it is much more difficult to bull through the logical issues.
1Oh, and the time. Where does the time go? Most of the solutions to the world’s problems are known. They are just too far down on anyone’s to-do list.
2While not the best example, the one that comes to mind is Fermat’s Last Theorem. You can Google the specifics, but I remember on the show Star Trek: Next Generation, which takes place in the years 2364-2370, the captain muses that “like Fermat’s Theorem” perhaps this puzzle will never be solved. About a year after the series wrapped, Fermat’s Last Theorem was resolved, making the episode an anachronism.
3Now maybe your organization is lucky or thoughtful, and you run your Culture of Safety or medical staff survey at the same time. Good on you. Most organizations spread them out because it is easier to manage them one at a time rather than all at once.
PX Matters with Joseph Snipp 
Leave a comment