One day last fall, I was talking with a colleague about a question that had come up from our partners in Baltimore’s performance management program, CitiStatSMART. Those folks were looking to identify key performance indicators (KPI) to use with certain city departments, but they had a problem: there were too many possible measures. Because the city had done performance management before, there were measures from that system. There were also measures used by the budget office in their outcome-based budgeting evaluating service delivery. Plus, there were proposed measures that had come from a survey of department staff requested by CitiStatSMART. For the police department, for example, that added up to 176 items!

My colleague had been working with a city that was in a similar situation, and this seemed like the kind of situation that other cities might encounter. Was there anything GovEx could do to help? We started from a conviction that for performance measures, more is not better. After all, if you track everything, then you pay attention to nothing–the important data can get lost in a sea of noise. What’s needed are a few KPI, and a strong commitment to regular review of the data to inform discussion, driving toward decisions that improve service delivery.

But how to select those few KPI from a long list of contenders, and stay within our schedule?  We began by making a list of the things we would want to know about a data point to evaluate it as a potential KPI. We came up with the following (see glossary linked below for definitions):

1. Is data available? [Yes/No]
2. Does the city own the data? [Yes/No]
3. Data refresh frequency? [Daily, Monthly, Quarterly, Annually]
4. Is this a previously established KPI? [Yes/No]
5. Estimated Level of Effort to Obtain [1 easy – 3 difficult]
6. Confidence in Data Quality [1 – high, 3 – none]
7. Measure Type    [Input, Output, Service Quality, Efficiency, Outcome]
8. Relation to Strategic Priorities [Direct / Indirect]
9. Responsive to your efforts? [Yes/No]
10. Interpretable by laypeople? [Yes/No]
11. Mutually agreed? [Yes/No]

But we soon realized that answering these questions for every measure would be a huge lift. Taking a second look, we saw that you don’t need to answer all the questions– just enough to exclude measures that don’t measure up. If a measure is on the list but data is not available for it, then we don’t need to go any further down the list. If a measure has data, but it’s not owned by the city (e.g. data from the State police) or the data is only available annually, then we can exclude that one too.

As we continued with this new process, things began to look a lot more workable. By separating the questions into three levels, we could get the information we needed (“is this a good KPI candidate or not?”) with a lot less effort. Those levels were:

We made a spreadsheet template and accompanying glossary that allowed a city to paste in a list of measures, and then use that to structure a meeting to review those measures.

This all seemed pretty cool on paper, but we needed to put it to the test. We circled back with our partners in Baltimore, and talked through how this might work for them in their upcoming meeting with the police department. They saw the benefit immediately, and made a couple of modifications to suit their needs. Then we set up three meetings over the next two weeks: one each with representatives from the Police Department, the Department of Transportation, and the Department of Public Works.

In each meeting, we explained the concept, and got down to work, talking through each measure and answering enough questions to leave it on the list, or cut it. In that session, a couple of lessons became clear:

• We weren’t going to make decisions in the room: Leadership saw this as an opportunity to collect valuable information, but not the right time to finalize anything.
• It helped to have a data inventory: The conversations went much more smoothly if the department had a good handle on its data systems. Many of the potential KPI were data points from systems listed in a data inventory. If a department knew what it had, where it was located, and what condition it was in, that was a great start.
• We needed to have all the right people in the conversation: Answering certain questions required an intimate knowledge of the data system, and for that we needed some folks who were not present.
• It was important to respect the history: This had all started because there had been previous versions of performance management, and it was unrealistic to think that we could wipe the slate clean. Department staff had concerns related to previous work in this area, and frustrations that they wanted to express. That was part of the process.
• Alignment with the budgeting process was important: Baltimore collects performance data as part of the budgeting process, and the departments were keen to be sure that measures used for performance management aligned to the budget process.
• This was the beginning of a longer process: Listing and evaluating many potential performance measures began a conversation between the performance management team and the department folks, and within the department as well. There were lots of follow-up meetings, and work to do between meetings. Also, this effort focused on performance surfaced questions related to data management.
• This had huge potential: As a result of this methodical approach, we were learning so much about the data systems in each department, and building a lot of trust with each staff. As we went along, department folks were able to speak candidly about areas of potential improvement, and express their opinions about the best measures to use.

Following up on those initial meetings, CitiStatSMART staff followed up with each department many times, convening more meetings and working sessions to collect all the data on measures, and to iron out the decisions on which measures were best. The slow and steady approach was worth it. Recently, CitiStatSMART was able to sign MOUs with two of the departments, laying out exactly how performance management will work for each of them, and listing those tested, agreed-upon performance measures.

Remember how it was stated above that the Police Department began with 176 KPIs? Through our process, they were able to reduce them to a very doable 27 measures! This new process will help every government in a similar situation. Use the 11 questions above to help you reduce the quantity of your KPIs so that your time can be better used towards working together to identify problems and solutions.