Content Discussion History

How Do We Measure Up? Do's and Don’ts for Benchmarking Your Data

When organizations begin to dig into the data about their service delivery, one of the first questions they ask is “Well, how do we compare to others?” This question makes a lot of sense! We want to put the data to work for us. One way to give data context is to make comparisons to similar organizations, a practice known as “benchmarking.” Benchmarking can be valuable, if done well.

To do it right, we need to ask ourselves: to what sources should we compare our data? And what conclusions can we draw from these comparisons?

When selecting an organization to serve as a point of comparison, look for two things:

  1. An organization that is similar, in the ways that matter. An organization might be similar in terms of scale (as measured by annual operating budget, for example), but is it similar in terms of the scope of its work? If your organization handles services for people experiencing homelessness, and also affordable transitional housing, it may be tough to compare against an organization that only handles one of those functions. Even if you can break out your data for those two functions, your performance is affected by the fact that you do both things under one roof. Find organizations whose work is similar to yours.
  2. An organization whose data is good, and similarly structured. Every organization has to work to ensure data quality, so don’t assume that others have great data. Instead, you’ll need to understand their data quality limitations (more on that below) and go into things with eyes open. Regarding structure: Imagine that you have been collecting income data at the household level, and they have collected it at the individual level. That’s not going to be a good match.

No organization will be exactly the same as yours – the goal is to identify the most defining aspects of your organization, and look for other organizations that are somewhat similar. 

To evaluate the criteria above, you’ll have to set up some in-depth conversations with potential partners, including leadership, subject-matter experts and data professionals. Make sure to discuss how they structure their work and how they calculate data points you want to use as benchmarks. In talking to them you may find that they have little confidence in a certain data point, the data point is influenced by seasonal factors, or the data point has changed substantially recently for some reason. You’ll want to know about all those circumstances, so that you can properly compare your data to theirs. It’s useful to think about the “Yeah, but. . .” factors that are attached to our own datasets and assume that everyone else has similar things going on.

Here’s a list of questions to ask a potential benchmarking partner:

Questions about the organization

Questions about their data

  • What services do you offer?
  • Who do you consider to be your client?
  • How do you define successful outcomes for clients?
  • How do you define success for the organization?
  • In what contexts or communities do you work?
  • What resource constraints do you face, and how do you adapt to them?
  • What are the biggest challenges to your service delivery?
  • What areas of service delivery are most successful?
  • Where do you invest most of your resources?
  • What data collection method drives this data?
  • How confident are you in the data collection?
  • How long have you collected data on this? What has changed in the way that you collect this data?
  • Is this the raw data? Is it cleaned or audited?
  • How is this data transformed from collection to reporting?
  • What data standard applies to this data?
  • Is this data collected for a specific purpose?
  • How is this data reported / published?
  • Have you ever analyzed this data? If so, what trends / patterns have you identified? What conclusions have you drawn?
  • What caveats do you have when you think about using this data?
  • Would you use this data to measure your own performance?

It’s a long list of questions, but that’s a good thing. We want to really understand our prospective partner and their data before we use it. These factors can and should show up in our analysis as reasons why a particular comparison is more or less valid. The analysis should always keep in mind that there are many factors that influence performance, and that we have data on only some of them.

We can also reflect on the answers to these questions as they relate to our own data. Having an in-depth, two-way conversation can help to form a relationship, which is a great thing if you’re looking to sustain this benchmarking over time. As you identify good matches, make sure to plan how you will use this analysis. If the comparison reveals areas where you outperform the partner, would you invest more in that area of success, or would you shift resources to a more challenged area? What about if the comparison reveals underperformance? Build a process for your leadership to reflect on what you will learn so that any surprises don’t spur people into hasty decisions.



Comments

add comment

Your comment will be revised by the site if needed.