Assessing the Quality of Performance Information or Performance Reports
Quality refers to the relevance and reliability of performance information and to the adequacy of performance reports with respect to meeting user needs.
Practice 2a. Test relevance or reliability: Test or certify performance measurement relevance, reliability, or both.
In testing relevance, for example, the auditor may test whether measures are pertinent, inclusive, timely, and understandable. In testing reliability, the auditor may test whether measures are accurate, impartial, correctly defined, and consistent. The auditor may also certify or attest to a level of relevance and/or reliability, or otherwise issue an opinion on relevance and/or reliability, according to externally- or internally-set standards or criteria. In addition to auditing standards, such as those promulgated by The IIA, the U.S. GAO, or CICA (in Canada) attestation standards may be used to guide or support this practice. In the United States, attestation standards issued by the AICPA may be used.
The Austin, Texas, Office of the City Auditor (OCA) has been reviewing City departments’ performance measures since 1985. For more recent reliability testing, OCA has developed guidance documents and has been using data mining software for less reliance on sampling, and GIS software for testing geographic-based performance measures … Read more
The Austin, Texas, Corporate Internal Auditor’s Office has been conducting formal performance certification reviews using its 17-step process since 2002. Corporate Internal Audit has also developed a process by which department staff can assess the accuracy of their own performance data and system controls. … Read more
In British Columbia, the Office of the Auditor General focused on relevance of performance measures and prepared a discussion paper on the topic. The paper explores how to audit the relevance of performance measures using 3 criteria and also discusses the applicability of direct audits versus attestation for assessing relevance. … Read more
The Colorado Springs, Colorado, City Auditor conducted several years of multi-department audits of measures entailing several mechanisms for ensuring relevance and reliability as well as comparability and cost-effectiveness. This has helped tighten up performance measurement so that the emphasis for verifying performance measures is now on specific performance audits, where deemed appropriate. … Read more
In Florida, the Office of Program Policy Analysis and Government Accountability’s (OPPAGA) performance measurement systems have matured under its performance based budgeting system. After issuing “accountability ratings” to State programs during the 1990’s, OPPAGA began reviewing programs for continuing relevance and reliability, a practice that continues with its 2006 “Sunset Review” program. … Read more
The Kansas City, Missouri, City Auditor’s method of testing relevance and reliability criteria include considering comparability with other organizations that provide the same service, and determining whether data collection personnel know why they are collecting data in a specific way. … Read more
The Louisiana Legislative Auditor’s staff examined a sample of performance indicators of state programs quarterly, and verified the material and substantive accuracy of information in agency performance incentive reward proposals. Audit staff assessed risk to pick measures to test, and followed a fieldwork plan to test reliability and relevance. … Read more
In Maricopa County, Arizona, Internal Audit adapted the Texas certification system for its own use to fulfill its role in the Managing for Results Policy passed by the County Board of Supervisors. Internal Audit reviews several risk factors to decide which measures to test. Internal Audit certifies reliability, and they developed a self-assessment checklist for departments to review strategic alignment (relevance) of their own measures. … Read more
The Nevada Legislative Auditor’s approach to testing agencies’ performance measure reliability includes assessing the reasonableness of measurement methodologies. Recent reliability testing of performance measures in the budget has increased legislators’ interest in agencies’ measures and led to management issuing guidance on ensuring reliability . … Read more
The state of Oregon began an integrity auditing program to test the accuracy and reliability of participating agencies’ Key Performance Measures. The first statewide series of integrity audits emphasized whether data can be verified, and resulted in four major types of findings for agencies to learn from to improve performance data. … Read More
In 2000-01, the Phoenix, Arizona, City Auditor Department validated all measures and how they are graphically presented in the monthly City Manager Executive Report to prepare for publication of the report on the Internet; now the Auditor Department validates about 25 percent of measures in the report each year, and reviews all new measures and graphs before they are posted. … Read Example from 2004 Guide (PDF)
The Portland, Oregon, City Auditor’s Office annually reviews data and backup documentation submitted by management for all measures in Portland’s annual Service Efforts and Accomplishments (SEA) Report. To test reliability, auditors examine all supporting documentation and the pattern of the data and interview agency staff about identified data concerns. Relevance is mainly reviewed for new measures. … Read more
In Prince William County, Virginia, Internal Audit annually audits about five to ten percent of measures the County reports in the annual budget and SEA Report. Internal Audit follows a standard set of steps and focuses on accuracy of performance data. … Read more
The Texas State Auditor pioneered testing performance measurement reliability. The Auditor’s Office uses a ten-step process to certify the accuracy of performance measures which has become a model adapted by other state and local auditors. … Read more
The Virginia Auditor of Public Accounts annually reviews the completeness and reliability of agency performance measures. Performance data security has been strengthened through this process, and the practice has evolved so that each audit may focus on some element of the system that may need improvement. These reviews are now done in the context of a revised state system that more strongly aligns performance measures with the budget, strategic planning, and long-term societal and quality of life indicators . … Read more