Practice 2a.Test relevance or reliability: Test or certify performance measurement relevance, reliability, or both.
As part of the process of compiling information to be included in the City of Portland’s Service Efforts and Accomplishments Report, each year, the City Auditor’s Office reviews the relevance and reliability of the measures to be included in the report each year. In 2006, this covered a total of twelve major City services, representing 76 percent of the City’s budget and 83 percent of its staff.
The City Auditor’s Office annually sends a data collection form to all agencies included in the SEA Report each year in order to request the most recent year’s data, backup documentation, and a contact person. The Auditor’s office reports that the annual collection forms and the frequent contacts with auditors who work individually with each Bureau reinforces the importance of the whole performance measurement exercise by helping them focus on performance measures through much of the year, rather than as an afterthought.
As SEA reporting has matured, the need to test relevance has diminished in part because changes in performance measures have been incremental in recent years, with only a small number each year. To test reliability, auditors examine all of the supporting documentation and the pattern of the data, looking for unsubstantiated or questionable results, and ask agency staff probing questions about data concerns identified in that review. The level of time spent and depth of the review process may vary based on prior agency history, staff turnover, or the presence of new measures. On average, audit staff conduct more in-depth reviews of two or three critical measures each year, depending upon management’s responses to questions. In the course of performance audits, auditors may also occasionally review the relevance and reliability of agencies’ more detailed performance indicators beyond the measures included in the SEA reports.
More detailed information on testing relevance and reliability in Portland, including additional reasons why auditors may still test relevance or examine reliability in more depth [Example in 2004 Guide-PDF]
Service Efforts and Accomplishments 2005-06 (PDF 2.7MB) This report, as well as previous years’ Service Efforts and Accomplishments reports are available as “publications” from the City Auditor’s website or as “audit reports” from the Audit Services Division home page.