RESULT BASED PERFORMANCE MEASUREMENT/MANAGEMENT STUDY
Site Visit Summary
Partnership for Reinventing Government
Date: May 5, 1999
FDEP Representative: Darryl S. Boudreau
SITE REPORTER: Audrey Borja, BMS Team
Other team members present:
We interviewed the Program Administrator for Strategic Projects and Planning, who gave a thorough and enthusiastic presentation about the results based performance measurement/management system of the agency.
For the past several years, the growth in environmentally regulated entities has greatly exceeded the growth of environmental regulatory agency budgets. In addition, the impacts that are having the largest effect on the environment have broadened to include unregulated impacts such as non point source runoff. As a result, it is becoming increasingly difficult for environmental agencies to make environmental gains using strictly traditional approaches.
In response, the FDEP began to focus and manage for results by building a performance measurement system that identifies emerging problems and supplies the contextual information needed to target resources and build collaborative partnerships with outside stakeholders to fix them. The measurement system is embodied in the "Secretary’s Quarterly Performance Report" (SQPR) which the FDEP distributes to all stakeholders via hard copy and the Internet. The FDEP’s measurement system is different than traditional "balanced scorecard" measurement systems in that customer and employee satisfaction are not included in the measurement system.
The goal of the FDEP’s measurement/management system is to identify environmental problems and the impacts that appear to be causing the greatest environmental harm as soon as possible so that roles for each stakeholder can be identified and implemented. Roles for regulatory agencies include such activities as addressing noncompliance and conducting research while the roles for outside stakeholders include such activities as lowering energy usage and the implementation of Best Management Practices.
The attached contact map shows the scope of the FDEP in working with other locations across the country and the SQPR communicates the FDEP programs, its intensive performance measurement system activities, and its outcomes to its stakeholders.
Highlights from interview include:
The future for FDEP leads them toward working with stakeholders to identify Florida’s environmental priorities; increasing Department accountability; improving its performance measurement system and streamlining reporting to EPA.
B. Process for establishing a set of measures
As noted previously, the FDEP’s performance measurement system does not attempt to measure customer or employee satisfaction. However, the FDEP does have a set of performance measures that focus on the outcomes they are striving to achieve such as healthy air and water, and supply the contextual information needed to evaluate changes in the observed outcomes (see framework below). Their outcome measures are exemplary. The Resource efficiency measures are evolving and are considered as critical to their overall success.
The seriousness of using a performance based approach to their agency began when their Secretary said – where do we put our resources, and what are our outcomes, what are we looking for? FDEP took the initiative and consulted with Malcolm Sparrow from Harvard University to assist in developing the best way to approach performance planning.
This resulted into a 4-Tier framework:
Due to the very short timeline in which the system was developed, the internal cultural change management component was under-addressed. The FDEP is moving from an activity to outcome based management system to protect the environment and public health, but the culture change is difficult. A major lesson learned from the FDEP experience was that cultural change management efforts need to begin as soon as or before the effort to develop new performance measures. Currently, the FDEP is using the information contained in the SQPR to manage for results and guide its result based management decisions.
C. Accountability for a balanced Set of measures
FDEP has several systems they are using to maintain accountability. The main one source document supporting their programs is the Secretary’s Quarterly Performance Report.
D. Measuring performance data collection and reporting
FDEP is continually improving their methods of measuring performance to get results in protecting public health and the environment. For example, with the generation of statistically valid compliance rates, the age-old enforcement question of "are enforcement numbers going up or down" is supplemented with a much more relevant question "are compliance rates going up or down". If compliance rates are increasing, than a decrease in enforcement numbers would be expected. However, if compliance rates are dropping as enforcement numbers drop, than there is a problem. This is the power of an integrated measurement system.
When environmental or public health problems are suspected, the FDEP utilizes a highly structured process called Environmental Problem Solving (EPS) to clearly identify the important problems and fix them. The steps are to:
A form of root cause analysis is an important part of the EPS. The FDEP identifies the factors having the largest impact on an identified problem and analyzes what can be done to lesson the impact of each factor. There are two levels of analysis: 1) consists of analyzing trends and patterns of data to identify the factors having the largest impact on important problems; and 2) takes the analysis a step further by determining what is causing each identified factor so that the appropriate integrated response can be designed – enforcement, compliance assistance, collaborative partnerships, etc.
In developing compliance rates, the FDEP first decided to put a definition on compliance, that is the distinction between "full compliance" and "significant compliance". Significant compliance is defined as the percentage of facilities with no violations that impacted or had the potential to impact human health or the environment. Significant compliance is used to ensure that those facilities with violations that have the potential to impact human health or the environment are highlighted, prioritized and dealt with in a timely fashion.
Another major decision in developing compliance rates was to reject the use of a "compliance status". It was proposed by some staff to use "the percentage of facilities in compliance at a particular point in time" as the equation to generate compliance rates. However that equation would result in a compliance status and not a rate. Since the compliance rates were to be used in resource targeting decisions by comparing facility sector compliance, a statistically valid rate was needed. Accordingly, the FDEP developed a method for generating statistically valid rates. The method uses representative and consensus sampling to generate valid rates.
For larger populations random sampling techniques can provide cost effective estimates of compliance. By placing inspection resources into sampling modes, regulatory agencies can learn more about the location and nature of noncompliance problems within different sub-populations of the regulated community.
E. Analyzing and reviewing performance data
FDEP is rethinking their information systems to capture a great deal of data. For example, they can now store compliance and enforcement functions and track data based on facilities in lieu of the activity conducted. Several components to their data systems have enhanced the factors available to make informed decisions.
The FDEP also uses data from other sources to supply needed contextual information. For example, per capita energy usage data are obtained from the Public Service Commission and Average Daily Vehicle Miles Traveled data are obtained from the Department of Transportation. These data points are critical to further understanding factors impacting the environment. For example, as citizens demand more energy and travel more miles in their automobiles, there is an increase in the amount of pollution emissions from power generation plants and from automobiles. Both of these increases in pollution may be in 100% compliance with environmental regulations and permits.
Each quarter, the data contained in the four tiers of information in the SQPR are analyzed and used to make management decisions by the identification of "Good", "Watch" and "Focus" areas.
F. Reporting to customers and stakeholders
Once the "Good", "Watch" and "Focus" areas have been identified, the FDEP issues Press Releases – which detail the Department’s findings. The press releases have significantly improved the FDEP’s relationship with the media and environmental groups. In conjunction with the press releases, over 1,500 copies of the SQPR are printed and distributed to the general public as well as placed on the FDEP’s Internet site. For improved accountability to stakeholders, the FDEP’s data systems are working toward an integrated facility based database system.
Transparent accountability is part of their daily work.
G. Linkages between strategic planning and resources allocations decision making, etc.
The "accountability" factor is not popular and is being addressed via the cultural change management mentioned above. However, FDEP has the strong support of the Secretary and the Deputy Secretary. The "Good", "Watch" and "Focus" areas mentioned above tie performance measurement to performance management and strategic planning since resources are redirected to address problems identified by "Watch" and "Focus" areas.
The FDEP began their performance measurement work prior to most other environmental agencies and is viewed as the leader in performance measurement for environmental agencies. As such, very little benchmarking has been conducted. The FDEP has worked closely with the Environmental Protection Agency’s Office of Enforcement and Compliance Assurance during the development of their National Performance Measurement Strategy.
Lessons learned and other
The FDEP believes that the best method for developing performance measures is to focus on the results that you are striving to achieve (for them it is the protection of the environment/public health) then identify the contextual information needed to analyze changes in the observed results. As the information is collected, it is vital that the information be analyzed, used to make resource targeting decisions and distributed to stakeholders.