Open Data: Measuring What Matters

Introduction

Open data and performance analytics are often separate practices, but are inextricable when it comes to the operations of modern governance. Open data provides the raw material for analysis and open data programs can have many goals, including increasing government transparency, sharing information to aid in decision making, engaging a broader coalition of stakeholders, and increasing government accountability. Performance analytics, on the other hand, describes the set of tools used to carry out analysis of government data and performance. While performance analytics is most often used to gain insight into a city’s progress toward strategic priorities, it should also be used to measure the performance of the open data program itself. Just as many transportation, police, and public works departments track a variety of metrics, information technology departments should track their open data programs as well to ensure that they are achieving their intended outcome.

The following is a framework and list of metrics to help your city measure the performance of your open data program. It is based on the City of San Francisco’s Evaluation and Performance Plan, the World Wide Web Foundation’s open data common assessment framework, and the U.S Department of Transportation’s Open Government Plan. The framework focuses on four areas: compliance environment, data, use, and impact. For more assistance with defining goals and expectations, see our Setting Performance Targets: Getting Started Guide.

Compliance Environment

Your open data environment describes the requirements you are obligated to fulfill according to your local open data policy, strategic priorities, state requirements, or federal requirements. Measuring your environmental performance can be as simple as tracking whether or not you meet the explicit policy requirements. Answering the following questions can help you determine your compliance environment:

  • What does our local open data policy require?
  • What do our state/federal grants require us to report?
  • What do the state/national organizations we belong to require us to report?
  • What do the state/federal laws require us to report?
  • What data do we collect related to our strategic priorities?

For example, a municipal open data policy may require that the city develop and release a program guide or produce quarterly reports on the program.
Open Data Qualitative Measures

 

Milestone Status Frequency
Implement the open data policy On Schedule, Delayed, Completed Weekly, Monthly, Quarterly
Encourage innovation through public engagement
Improve access to data through visualization platforms
Coordinate geospatial data efforts
Report on legislative implementation

Data

Data metrics are directly related to the data assets that you have and publish, encompassing the structure and quality of your data. Lots of work goes into inventorying and releasing data, and it’s incredibly useful to keep track of this process. The following metrics describe a city’s plan for and progress on inventorying, prioritizing, publishing, and updating datasets.

Measure(s) Desired Trend / Target Frequency
Percent/# of departments that have completed the dataset inventory 100% Monthly, Quarterly
Percent/# of departments that have completed publishing plans 100% Monthly, Quarterly
Percent of inventoried datasets that are published Increase over time Weekly, Monthly
Percent of inventoried datasets published by dataset classification 100% for public data Weekly, Monthly
Percent of eligible datasets that are natively hosted Increase over time / 90% Weekly, Monthly
Percent of datasets updated on time 100% Daily
Percent of eligible datasets without a publishing lag 100% of eligible Daily
Median publishing lag in days for datasets with a publishing lag Less than 30 Weekly, Monthly
Percent of datasets with required metadata 100% Weekly, Monthly

Use

Use metrics describe how your open data is handled internally in your city, as well as externally within the community. Your city can conduct surveys to measure users’ perceived value of open data, as well as measure how often people are using certain datasets. This information can provide you with potential success stories and insight into what data the public is interested in.

Measure(s) Desired Trend / Target Frequency Collected Internal / External
Internal changes in behavior / circumstances
Respondents indicating that open data has made their analytical work:
  • easier
  • faster
  • more accurate
Increase over time Annual Internal
Percent of respondents indicating that they use the data portal to access data internally and from other departments Increase over time Annual Internal
Respondents indicating that the following are medium or major barriers
  • Knowledge/awareness of datasets (inter and intra departmental)
  • Getting data out of information systems (inter and intra departmental)
  • Not knowing who to contact
  • No standard process
  • Data not available in digital form
Reduction over time Annual Internal
Respondents indicating that the following would be helpful or very helpful
  • Better access to data produced or collected by other departments
Reduction over time Annual Internal
Products made with open data
Number of products made with open data (by type) Increase over time Ongoing Both
Respondents indicating that open data was essential or important for creating their product >50% Ongoing Both
Indicators of use of open data
Creation of derived views (not by core team) Increase over time At least daily Both
Percent of users that are return users Maintain At least daily Both
Number of dataset downloads* Increase over time At least daily Both
Number of Data Portal accounts* Increase over time At least daily Both
Number of API hits (all sources)* Increase over time At least daily Both
Number of API token holders* Increase over time At least daily Both

* Raw Data

Data portal analytics can be accessed via portal features, or an analytics service such as Google Analytics. GovEx has compiled data for these metrics, which you can access in .CSV format here. The data describes weekly metadata from 20 Socrata Portals ranging from 2012 through 2015. Please reference the ReadMe file in this folder for more information about the data.

Impact

An open data program shouldn’t release data for the sake of releasing data. The end goal should be to improve the lives of residents. Measuring the impact of open data can be difficult, but there are a few areas in which the relationships can be measured with open data. The following are some potential metrics to measure civic engagement, government effectiveness, and economic development. In addition, your city can conduct surveys to measure users’ perceived value of open data, as well as measure how often people are using certain datasets. This information can provide you with potential success stories and insight into what data the public is interested in.

Measure(s)

 

Desired Trend / Target Frequency
Civic Engagement Weekly,
Monthly,
Quarterly
Number of city responses to open data suggestions Increase over time
Open data presence on social media Increase over time
Number of projects made with open data (by type) Increase over time
Number of participants at a community event related to data on open data portal (e.g., hackathon, strategic plan visioning, data release party, etc.) Increase over time
City Government Effectiveness
Number of ad hoc data requests Decrease over time
Number of FOIA requests Decrease over time
Respondents indicating satisfaction with city services Increase over time
Number of internal or intra-departmental data requests Decrease over time
Number of staff hours responding to data requests Decrease over time
Economic Development
Respondents indicated that open data was essential or important for creating their product Increase over time
Economic value generated by open data in public/private sector Increase over time