Practice 2a.Test relevance or reliability: Test or certify performance measurement relevance, reliability, or both.
Since 2002, the Auditor of Public Accounts (APA) has been statutorily required to review and report annually to the General Assembly and Governor on the performance measures that state agencies report to the Department of Planning and Budget (DPB). The Auditor is also required to review the accuracy of the management systems used to accumulate and report the results and make recommendations for new or revised measures. Since then, APA auditors have been testing data completeness and reliability for a sample of state agency performance measures each year.
In 2006, the Commonwealth of Virginia reorganized its performance measures and introduced a multi-level system and new website that reports societal and quality of life indicators, along with agency performance measures subject to APA audit. Legislation driving these changes, since 2003, charged the new Council on Virginia’s Future with defining long-term state objectives and integrating agency performance into an overall framework of measures that support the objectives. As part of these changes, Virginia also implemented some changes in their programmatic budget structure which allows a link between the budget and strategic planning. During this process, the current Governor has identified 200 performance measures that are “key measures” for his administration. This new framework has changed the context of the performance measures and systems that APA audits.
The primary objective of each audit is to test performance measures to determine whether the information is complete, accurate, and reliable. In addition, the audit evaluates the reasonableness of performance targets. However, the practice has evolved in that each audit generally may include a special focus on some element of the system that may need improving. For example, prior audits have also focused on relevance of measures, understandability, user friendliness, and security of system data.
The number of hours allocated to reviewing performance measures has increased as the APA has added additional audit procedures and selected additional measures as they have changed over the years. The fiscal year 2006 multi-agency audit of performance measures is budgeted at 750 hours. APA is also planning to increase resources to address performance standards in colleges and universities (which APA previously reviewed in selected institutions in 1999). The APA reports that through the use of this practice, security over performance measure data has been strengthened. Budget analysts now perform reasonability checks on data prior to publishing the data to the public website, and the Department of Planning and Budget has increased training and assistance it provides to agencies in preparing performance measures. However, APA continues to see weakness in understandability and consistency of measures.
More information on the legislative background for APA’s use of this practice, and APA’s approach to this practice before 2002 [Example in 2004 Guide-PDF]
- Review of Performance Measures for Fiscal Year Ended June 30, 2005 (PDF). This report and reports on prior years’ performance measure reviews are available as “reports” from the APA website.
- Review of Performance Measures at Selected Higher Education Institutions for the Year Ended June 30, 1999 (PDF)