Open data and performance analytics are often separate practices, but are inextricable when it comes to the operations of modern governance. Open data provides the raw material for analysis and open data programs can have many goals, including increasing government transparency, sharing information to aid in decision making, engaging a broader coalition of stakeholders, and increasing government accountability. Performance analytics, on the other hand, describes the set of tools used to carry out analysis of government data and performance. While performance analytics is most often used to gain insight into a city’s progress toward strategic priorities, it should also be used to measure the performance of the open data program itself. Just as many transportation, police, and public works departments track a variety of metrics, information technology departments should track their open data programs as well to ensure that they are achieving their intended outcome.
The following is a framework and list of metrics to help your city measure the performance of your open data program. It is based on the City of San Francisco’s Evaluation and Performance Plan, the World Wide Web Foundation’s open data common assessment framework, and the U.S Department of Transportation’s Open Government Plan. The framework focuses on four areas: compliance environment, data, use, and impact. For more assistance with defining goals and expectations, see our Setting Performance Targets: Getting Started Guide.
Your open data environment describes the requirements you are obligated to fulfill according to your local open data policy, strategic priorities, state requirements, or federal requirements. Measuring your environmental performance can be as simple as tracking whether or not you meet the explicit policy requirements. Answering the following questions can help you determine your compliance environment:
- What does our local open data policy require?
- What do our state/federal grants require us to report?
- What do the state/national organizations we belong to require us to report?
- What do the state/federal laws require us to report?
- What data do we collect related to our strategic priorities?
For example, a municipal open data policy may require that the city develop and release a program guide or produce quarterly reports on the program.
Open Data Qualitative Measures
|Implement the open data policy||On Schedule, Delayed, Completed||Weekly, Monthly, Quarterly|
|Encourage innovation through public engagement|
|Improve access to data through visualization platforms|
|Coordinate geospatial data efforts|
|Report on legislative implementation|
Data metrics are directly related to the data assets that you have and publish, encompassing the structure and quality of your data. Lots of work goes into inventorying and releasing data, and it’s incredibly useful to keep track of this process. The following metrics describe a city’s plan for and progress on inventorying, prioritizing, publishing, and updating datasets.
|Measure(s)||Desired Trend / Target||Frequency|
|Percent/# of departments that have completed the dataset inventory||100%||Monthly, Quarterly|
|Percent/# of departments that have completed publishing plans||100%||Monthly, Quarterly|
|Percent of inventoried datasets that are published||Increase over time||Weekly, Monthly|
|Percent of inventoried datasets published by dataset classification||100% for public data||Weekly, Monthly|
|Percent of eligible datasets that are natively hosted||Increase over time / 90%||Weekly, Monthly|
|Percent of datasets updated on time||100%||Daily|
|Percent of eligible datasets without a publishing lag||100% of eligible||Daily|
|Median publishing lag in days for datasets with a publishing lag||Less than 30||Weekly, Monthly|
|Percent of datasets with required metadata||100%||Weekly, Monthly|
Use metrics describe how your open data is handled internally in your city, as well as externally within the community. Your city can conduct surveys to measure users’ perceived value of open data, as well as measure how often people are using certain datasets. This information can provide you with potential success stories and insight into what data the public is interested in.
|Measure(s)||Desired Trend / Target||Frequency Collected||Internal / External|
|Internal changes in behavior / circumstances|
|Respondents indicating that open data has made their analytical work:
||Increase over time||Annual||Internal|
|Percent of respondents indicating that they use the data portal to access data internally and from other departments||Increase over time||Annual||Internal|
|Respondents indicating that the following are medium or major barriers
||Reduction over time||Annual||Internal|
|Respondents indicating that the following would be helpful or very helpful
||Reduction over time||Annual||Internal|
|Products made with open data|
|Number of products made with open data (by type)||Increase over time||Ongoing||Both|
|Respondents indicating that open data was essential or important for creating their product||>50%||Ongoing||Both|
|Indicators of use of open data|
|Creation of derived views (not by core team)||Increase over time||At least daily||Both|
|Percent of users that are return users||Maintain||At least daily||Both|
|Number of dataset downloads*||Increase over time||At least daily||Both|
|Number of Data Portal accounts*||Increase over time||At least daily||Both|
|Number of API hits (all sources)*||Increase over time||At least daily||Both|
|Number of API token holders*||Increase over time||At least daily||Both|
* Raw Data
Data portal analytics can be accessed via portal features, or an analytics service such as Google Analytics. GovEx has compiled data for these metrics, which you can access in .CSV format here. The data describes weekly metadata from 20 Socrata Portals ranging from 2012 through 2015. Please reference the ReadMe file in this folder for more information about the data.
An open data program shouldn’t release data for the sake of releasing data. The end goal should be to improve the lives of residents. Measuring the impact of open data can be difficult, but there are a few areas in which the relationships can be measured with open data. The following are some potential metrics to measure civic engagement, government effectiveness, and economic development. In addition, your city can conduct surveys to measure users’ perceived value of open data, as well as measure how often people are using certain datasets. This information can provide you with potential success stories and insight into what data the public is interested in.
|Desired Trend / Target||Frequency|
|Number of city responses to open data suggestions||Increase over time|
|Open data presence on social media||Increase over time|
|Number of projects made with open data (by type)||Increase over time|
|Number of participants at a community event related to data on open data portal (e.g., hackathon, strategic plan visioning, data release party, etc.)||Increase over time|
|City Government Effectiveness|
|Number of ad hoc data requests||Decrease over time|
|Number of FOIA requests||Decrease over time|
|Respondents indicating satisfaction with city services||Increase over time|
|Number of internal or intra-departmental data requests||Decrease over time|
|Number of staff hours responding to data requests||Decrease over time|
|Respondents indicated that open data was essential or important for creating their product||Increase over time|
|Economic value generated by open data in public/private sector||Increase over time|