A version of this article originally appeared in GAP News (3rd Quarter, 2007),The Institute of Internal Auditors’ e-newsletter for government auditors ). Reprinted with permission.
By testing performance measures, auditors can complement a regular performance auditing program and improve government performance and accountability.
PAUL EPSTEIN, Principal
EPSTEIN & FASS ASSOCIATES
ALINA SIMONE, Associate Consultant
EPSTEIN & FASS ASSOCIATES
Download the complete GAP News article (PDF)
Performance auditing of government services is a widely accepted practice that has been in use for at least four decades. Through performance auditing, auditors add value to government performance by recommending improvements in the economy, efficiency, and effectiveness of programs. A number of high-leverage roles and practices concerning how government auditors relate to performance measurement also add value to performance and accountability, and some of these practices can increase the value of performance audits. The main focus of this article is on auditing or testing the relevance and reliability of performance measurement. Leaders in the field have found testing performance measures and data to be a faster practice to conduct than performance auditing, and, sometimes, a more far-reaching audit tool that complements a regular program of performance auditing.
Auditing Performance Information as Distinct From Performance Auditing
Performance auditing is one practice in the first role of the Framework of Auditor Roles and Practices in Performance Measurement. But the framework goes well beyond that. A useful starting point for auditors to enhance the ways they improve government performance and accountability is by assessing the quality of performance information or performance reports, which is Role 2 of the framework. This is the first of a series of articles on auditors using roles and practices of the framework to add value to government performance. The first of three practices of Role 2 is particularly important — Practice 2a: Test Relevance or Reliability: Test or certify performance measurement relevance, reliability, or both.
The relevance of performance information refers to how well performance measures are aligned with goals and objectives, and how useful they are for accountability and for decision making by management, elected officials, and citizens. Reliability refers to how well defined current performance measures are, and whether data for the measures are accurate and precise enough for decision making and accountability.
Testing performance information is sometimes done as part of an agency performance audit. But some audit offices have found it valuable to test performance measures as separate audit projects in themselves, on the premise that no matter how well agencies are run, their accountability and ability to improve will be hampered by a lack of useful performance information. These “performance information audits” can be done considerably faster, and involve less auditor time (e.g., 25-40 staff hours per measure to test reliability, much less to test relevance), than full performance audits of an agency’s operations. Total time used depends on how many performance measures an audit office decides to test. Some audit offices, following the example of the Texas State Auditor, assign each measure they test to a “certification category” to clarify the status of an agency’s measures and needs for improvement.
Capturing Value Added
Auditors’ testing of performance measures adds value to the government organizations they serve by helping agencies make higher quality performance information available. Government organizations capture that value by using the information to improve accountability, decision making, and managing for results.
By assuring the quality of performance information, auditors also add value to their existing performance audit practices. When agencies have relevant, reliable performance data, auditors don’t have to spend a lot of time collecting primary data to assess a program or an organization’s basic performance, which frees up auditor time for higher-value activities. Audit offices have captured this value in several ways:
- By conducting faster, more efficient performance audits. This can happen when an audit office becomes a user of reliable performance information, as Florida’s Office of Program Policy Analysis and Government Accountability (OPPAGA) has become. For example, OPPAGA used the measure of the food stamp error rate to find that high food stamp errors cost the state $2.2 million for Fiscal Years 2001 and 2002.
- By conducting more thorough performance audits, such as deeper evaluations of impact on the population served. OPPAGA does this when the legislature wants an evaluation of policies or programs whose effects are not apparent from regularly reported agency performance data, as it did in an evaluation of Bright Futures scholarships. After analyzing databases on student test scores and transcripts, OPPAGA concluded that, although other factors may have contributed, students who were eligible for the Bright Futures scholarships took more demanding courses and performed better than students who graduated before the scholarships became available.
- By using reliable performance information as part of an organization-wide risk assessment to pick future performance audits. The Austin City Auditor does this. In the risk assessment model used by the Office of the City Auditor (OCA), performance trends are a part of a planning and performance risk factor, which represents 20 percent of total risk factors — the highest-weighted factor in the model. OCA considers increases and decreases in performance, how performance compares with external benchmarks (when available), and whether adequate meaningful performance measures are in place to help the OCA find higher-value targets for its regular program of performance auditing.
Once the quality of performance information has significantly improved, the audit organization may then find it useful to reduce its level of effort testing relevance and reliability, and to focus more on other practices, for example, helping agency managers make better use of performance data (i.e., part of Role 4 – Encourage or Assist Management).
Who’s Doing It?
The Texas State Auditor, the Austin City Auditor, and Florida’s OPPAGA are not the only audit offices that have audited performance information. This website features examples of 14 local, state, and provincial audit organizations across North America that have been testing relevance and reliability of performance information are posted on the. For the related practice of assuring performance reports, examples of five audit organizations are provided.
What Audit Tools Are Available?
Auditors across North America have provided data collection instruments, methodologies for practices, guidance papers, audit criteria, audit steps, and full audit programs for the “Auditor Toolkit” on this website. You can download these tools and adapt them to use in your entity. There are a particularly large number of tools posted for Role 2 — Assess Performance Information.
Training to Learn These Roles and Incorporate Them Into an Audit Practice
Several training courses are available to help auditors learn roles and practices related to performance measurement and incorporate them into their work. The course “A New Service Model: Auditor Roles in Government Performance Measurement” can help auditors determine their best opportunities for adapting new practices to use in the government organizations they serve. The courses “Assessing the Quality of Performance Information and Performance Reports” and “Assessing the Reliability and Relevance of Performance Information” are particularly focused on Role 2. Assess Performance Information. These courses were well received in 2007 and are being provided again in 2008 when on-site training will be available to audit organizations or professional groups. Additional courses will be available in the latter part of 2008 and in 2009. See training for currently scheduled courses, locations, dates, local contacts, and course descriptions. For more information, including how to schedule a course for your organization or group, send a message through the Contact Us page of this website.