Practice 2b. Assure performance reports: Audit, attest to, assure, or certify external performance reports.
The Budget Transparency and Accountability Act (BTAA), passed by the Legislative Assembly of British Columbia (BC) in 2000, highlights legislators’ desire for more meaningful and rigorous public scrutiny of government performance. The BTTA requires ministries, government organizations, and the government as a whole to provide three-year service plans and report annually on their performance against their plans. To help ensure the reporting of meaningful performance information, the Office of the Auditor General (OAG) assessed the quality of performance reporting contained in the annual reports of a cross-section of the government’s ministries and Crown corporations in each of the five years subsequent to the BTAA coming into force. Results of these assessments were published in the “Building Better Reports” Series (BBR). Initially, assessment results were published on an aggregated basis, although results were made available to individual organizations. Examples of best practice were highlighted so that organizations could learn from the practices of others, though over time this emphasis decreased in favor of the publication of individual assessments. Reports were not explicitly ranked in order of assessment scores, although interested organizations could easily see how their assessment compared to others. Each year, between 10 and 20 staff from across the office participated in assessment teams for periods ranging from 2 to 8 weeks, depending on the number of reports assessed by each team, with one assessment coordinator assigned for the duration of the project. In total, 3,000 staff-hours per year were devoted to this practice, including meeting with report preparers from each organization to assess results of OAG reviews with them.
Over the 5 years covered by the Building Better Reports series, performance quality was assessed using a “learning model” approach (see “tools” below) that showed progress over four stages of development for incorporating good reporting principles: “Start-up,” “In Process,” “Fundamentals in Place,” and “Fully Incorporated.” The OAG developed eight assessment scales, in matrix form, describing detailed characteristics for each of the four developmental phases for the following eight principles of a good performance report:
- Explain the public purpose served;
- Link goals and results;
- Focus on the few, critical aspects of performance;
- Relate results to risk and capacity;
- Link resources, strategies and results;
- Provide comparative information;
- Present credible information, fairly interpreted;
- Disclose the basis for key reporting judgments.
When the OAG assessed an organization’s performance report, it summarized the results in an assessment matrix showing the development stage for each performance reporting principle assessed. In 2003, after deliberation by a high-level steering committee, these eight principles were endorsed by the BC Legislative Assembly’s Select Standing Committee on Public Accounts (PAC) “as a basis for service plan and annual service plan report guidance and assessment.” The PAC also encouraged “the Auditor General and Government to revisit the principles at a future date to ensure that BC’s reporting principles continue to reflect good reporting practices in Canada and internationally.” Furthermore, a resolution was passed by the PAC to encourage “Ministries and Crown Agencies to work towards the goal of incorporating into their contracts with non-ministerial service delivery agencies the eight principles.” The OAG made the assessment scale matrices and methodology, as well as information on the PAC’s endorsement available to the public through use of both published and web-based reports. One such report, Reporting Principles and An Assurance Program for BC (March, 2003), also included self-assessment criteria that government organizations can use for assessing their own public reporting (see “tools” below).
From 2002–2006, the Auditor General used the same reporting principles as used in multi-organizational reviews to issue separate assurance reports on the annual performance reports of the British Columbia Public Guardian and Trustee (PGT). These include assessment of the relevance and reliability of the performance measures, and each project takes about 650 staff hours to complete. Similar assurance reports have been provided for the performance reports of the Workers’ Compensation Board of BC (WorkSafeBC) since 2004 and the British Columbia Assessment Authority in 2006. The OAG has received strong positive feedback in relation to its assurance engagements and WorkSafeBC, BCAA, and PGT. The reports of these three organizations are among the strongest in British Columbia’s public sector.
After five years, the OAG felt that the BBR series had begun to have less of an impact than they had initially, and that the quality of reporting had reached a plateau, especially in the case of the ministries. As a result, the OAG now devotes less time to the cross-government assessment of report quality, though it may become a priority again in the future. For 2006, the OAG is preparing a retrospective report examining the trends in performance reporting over the previous six years, and identifying the opportunities for improvement that remain.
Recognizing that learning phases must inevitably give way to a performance phase, in 2006 the OAG discontinued the “learning model” approach described above in favor of a “threshold approach” (See “tools” below). The quality of reports continues to be assessed on a principle-by-principle basis. However, rather than placing the results of these assessments on a rating scale, assessors are asked to conclude whether or not the principle has been fundamentally met. This standard is consistent with the expectations for a “fundamentals in place” assessment in previous learning model, and recognizes that the spirit and intent of each reporting principle can be achieved without 100% compliance with all of the detailed self-assessment criteria.
More information on the Building Better Report series and multi-organizational reviews [Example in 2004 Guide-PDF]
- Building Better Reports: Our Methodology for Assessing the Annual Service Plan Reports of Government (July, 2005) (PDF) which includes a discussion of the 8 principles of a good performance report and the 8 assessment scale matrices of the “Learning Model” for assessing reports.
- Reporting Principles and An Assurance Program for BC, (March, 2003) (PDF) which includes self-assessment criteria that government organizations can use for assessing their own public reporting.
- Threshold Approach – A New Approach for Assessing and Auditing Public Performance Reports, (2006 Working Draft) (DOC)
- Public Sector Governance: A Guide to the Principles of Good Practice; How are We Doing: The Public Reporting of Performance Measures in British Columbia (Dec 2008)
- Building Better Reports: Our Assessment of the 2004/05 Annual Service Plan Reports of Government, Report 10 (March 2006) (PDF) This report, as well as previous years’ reports from the “Building Better Reports” series back to 2001, and related reports and guides, are available on the BC Auditor General’s website, under “Publications.”
- “Report of the Auditor General of British Columbia on the Public Guardian and Trustee Performance Report” is in the Annual Reports section of the PG&T publications web page.
- Reports of the Auditor General of British Columbia on the Annual Report of the Workers’ Compensation Board of British Columbia (WorkSafeBC) are in the 2004 & 2005 Compensation Board Annual Reports at the Board’s Annual Reports webpage.
- “Report of the Auditor General of British Columbia on the 2005 Annual Report of the British Columbia Assessment Authority” is in the Authority’s 2005 Annual Report at the Assessment Authority’s reports webpage.