When organizations begin to dig into the data about their service delivery, one of the first questions they ask is “Well, how do we compare to others?” This question makes a lot of sense! We want to put the data to work for us. One way to give data context is to make comparisons to similar organizations, a practice known as “benchmarking.” Benchmarking can be valuable, if done well.
To do it right, we need to ask ourselves: to what sources should we compare our data? And what conclusions can we draw from these comparisons?
When selecting an organization to serve as a point of comparison, look for two things:
- An organization that is similar, in the ways that matter. An organization might be similar in terms of scale (as measured by annual operating budget, for example), but is it similar in terms of the scope of its work? If your organization handles services for people experiencing homelessness, and also affordable transitional housing, it may be tough to compare against an organization that only handles one of those functions. Even if you can break out your data for those two functions, your performance is affected by the fact that you do both things under one roof. Find organizations whose work is similar to yours.
- An organization whose data is good, and similarly structured. Every organization has to work to ensure data quality, so don’t assume that others have great data. Instead, you’ll need to understand their data quality limitations (more on that below) and go into things with eyes open. Regarding structure: Imagine that you have been collecting income data at the household level, and they have collected it at the individual level. That’s not going to be a good match.
No organization will be exactly the same as yours – the goal is to identify the most defining aspects of your organization, and look for other organizations that are somewhat similar.
To evaluate the criteria above, you’ll have to set up some in-depth conversations with potential partners, including leadership, subject-matter experts and data professionals. Make sure to discuss how they structure their work and how they calculate data points you want to use as benchmarks. In talking to them you may find that they have little confidence in a certain data point, the data point is influenced by seasonal factors, or the data point has changed substantially recently for some reason. You’ll want to know about all those circumstances, so that you can properly compare your data to theirs. It’s useful to think about the “Yeah, but. . .” factors that are attached to our own datasets and assume that everyone else has similar things going on.
Here’s a list of questions to ask a potential benchmarking partner:
Questions about the organization
Questions about their data
It’s a long list of questions, but that’s a good thing. We want to really understand our prospective partner and their data before we use it. These factors can and should show up in our analysis as reasons why a particular comparison is more or less valid. The analysis should always keep in mind that there are many factors that influence performance, and that we have data on only some of them.
We can also reflect on the answers to these questions as they relate to our own data. Having an in-depth, two-way conversation can help to form a relationship, which is a great thing if you’re looking to sustain this benchmarking over time. As you identify good matches, make sure to plan how you will use this analysis. If the comparison reveals areas where you outperform the partner, would you invest more in that area of success, or would you shift resources to a more challenged area? What about if the comparison reveals underperformance? Build a process for your leadership to reflect on what you will learn so that any surprises don’t spur people into hasty decisions.