Archive

RESULT BASED PERFORMANCE MEASUREMENT/MANAGEMENT STUDY

Site: National Partnership for Reinventing Government
Washington, D.C.

Date: May 5, 1999

FDEP Representative: Darryl S. Boudreau
Program Administrator
Strategic Projects and Planning, Office of the Secretary

SITE REPORTER: Audrey Borja, BMS Team

Other team members present:
Kathy Monahan, BMS Study Leader
Patty Sun, NPR
Gene Sheskin, USCS
Chris Tirpak, EPA

We interviewed the Program Administrator for Strategic Projects and Planning, who gave a thorough and enthusiastic presentation about the results based performance measurement/management system of the agency.

For the past several years, the growth in environmentally regulated entities has greatly exceeded the growth of environmental regulatory agency budgets. In addition, the impacts that are having the largest effect on the environment have broadened to include unregulated impacts such as non point source runoff. As a result, it is becoming increasingly difficult for environmental agencies to make environmental gains using strictly traditional approaches.

In response, the FDEP began to focus and manage for results by building a performance measurement system that identifies emerging problems and supplies the contextual information needed to target resources and build collaborative partnerships with outside stakeholders to fix them. The measurement system is embodied in the "Secretary’s Quarterly Performance Report" (SQPR) which the FDEP distributes to all stakeholders via hard copy and the Internet. The FDEP’s measurement system is different than traditional "balanced scorecard" measurement systems in that customer and employee satisfaction are not included in the measurement system.

The goal of the FDEP’s measurement/management system is to identify environmental problems and the impacts that appear to be causing the greatest environmental harm as soon as possible so that roles for each stakeholder can be identified and implemented. Roles for regulatory agencies include such activities as addressing noncompliance and conducting research while the roles for outside stakeholders include such activities as lowering energy usage and the implementation of Best Management Practices.

The attached contact map shows the scope of the FDEP in working with other locations across the country and the SQPR communicates the FDEP programs, its intensive performance measurement system activities, and its outcomes to its stakeholders.

Highlights from interview include:

Key Concepts:

• FDEP is building the capacity to conduct community based environmental management
• FDEP has no threats – all are involved; shared databases and action plans are shared efforts.
• There is a process of involvement at all levels: individuals and local, state, and federal governments

• Flexibility through measurement; discretion through delegation, and trust through accountability;
• Performance Partnership Agreement;
• Joint Compliance and Enforcement Plan – detailed plan to identify and address state/federal priorities;
• Identify issues and best tools to resolve them: root cause analysis
• Accountability – through Secretary’s Quarterly Performance Report
• Information
• Improvement one-stop integrated database

The future for FDEP leads them toward working with stakeholders to identify Florida’s environmental priorities; increasing Department accountability; improving its performance measurement system and streamlining reporting to EPA.

B. Process for establishing a set of measures

As noted previously, the FDEP’s performance measurement system does not attempt to measure customer or employee satisfaction. However, the FDEP does have a set of performance measures that focus on the outcomes they are striving to achieve such as healthy air and water, and supply the contextual information needed to evaluate changes in the observed outcomes (see framework below). Their outcome measures are exemplary. The Resource efficiency measures are evolving and are considered as critical to their overall success.

The seriousness of using a performance based approach to their agency began when their Secretary said – where do we put our resources, and what are our outcomes, what are we looking for? FDEP took the initiative and consulted with Malcolm Sparrow from Harvard University to assist in developing the best way to approach performance planning.

This resulted into a 4-Tier framework:

1. Environmental and Public Health Outcome Indicators – this Tier is the most important Tier and tracks the outcomes the Department is working for - clean air and water, safe drinking water, healthy fish, and unspoiled Florida to enjoy;
2. Behavioral and Cultural Measures – tracks the change we see in the way stakeholders behave toward the environment – the compliance rate of the various facilities, per-capita energy usage, volunteer hours spent improving ecosystems, adoption of Best Management Practices, common sense regulation efforts, etc.;
3. Departmental Outputs and Activities – this Tier covers what the staff does each day – number of inspections, violations, penalties assessed, etc.; and
4. Resources Efficiency – this is about financial accountability including accounting of the tax dollars with which they are entrusted to provide the outcomes of environmental protection (this Tier is the least developed)

Due to the very short timeline in which the system was developed, the internal cultural change management component was under-addressed. The FDEP is moving from an activity to outcome based management system to protect the environment and public health, but the culture change is difficult. A major lesson learned from the FDEP experience was that cultural change management efforts need to begin as soon as or before the effort to develop new performance measures. Currently, the FDEP is using the information contained in the SQPR to manage for results and guide its result based management decisions.

C. Accountability for a balanced Set of measures

FDEP has several systems they are using to maintain accountability. The main one source document supporting their programs is the Secretary’s Quarterly Performance Report.

D. Measuring performance data collection and reporting

FDEP is continually improving their methods of measuring performance to get results in protecting public health and the environment. For example, with the generation of statistically valid compliance rates, the age-old enforcement question of "are enforcement numbers going up or down" is supplemented with a much more relevant question "are compliance rates going up or down". If compliance rates are increasing, than a decrease in enforcement numbers would be expected. However, if compliance rates are dropping as enforcement numbers drop, than there is a problem. This is the power of an integrated measurement system.

When environmental or public health problems are suspected, the FDEP utilizes a highly structured process called Environmental Problem Solving (EPS) to clearly identify the important problems and fix them. The steps are to:

• Identify potential problem
• Define the problem precisely
• Determine how to measure impact
• Develop solutions
• Implement the plan with periodic monitoring and review
• Case closure

A form of root cause analysis is an important part of the EPS. The FDEP identifies the factors having the largest impact on an identified problem and analyzes what can be done to lesson the impact of each factor. There are two levels of analysis: 1) consists of analyzing trends and patterns of data to identify the factors having the largest impact on important problems; and 2) takes the analysis a step further by determining what is causing each identified factor so that the appropriate integrated response can be designed – enforcement, compliance assistance, collaborative partnerships, etc.

In developing compliance rates, the FDEP first decided to put a definition on compliance, that is the distinction between "full compliance" and "significant compliance". Significant compliance is defined as the percentage of facilities with no violations that impacted or had the potential to impact human health or the environment. Significant compliance is used to ensure that those facilities with violations that have the potential to impact human health or the environment are highlighted, prioritized and dealt with in a timely fashion.

Another major decision in developing compliance rates was to reject the use of a "compliance status". It was proposed by some staff to use "the percentage of facilities in compliance at a particular point in time" as the equation to generate compliance rates. However that equation would result in a compliance status and not a rate. Since the compliance rates were to be used in resource targeting decisions by comparing facility sector compliance, a statistically valid rate was needed. Accordingly, the FDEP developed a method for generating statistically valid rates. The method uses representative and consensus sampling to generate valid rates.

For larger populations random sampling techniques can provide cost effective estimates of compliance. By placing inspection resources into sampling modes, regulatory agencies can learn more about the location and nature of noncompliance problems within different sub-populations of the regulated community.

E. Analyzing and reviewing performance data

FDEP is rethinking their information systems to capture a great deal of data. For example, they can now store compliance and enforcement functions and track data based on facilities in lieu of the activity conducted. Several components to their data systems have enhanced the factors available to make informed decisions.

The FDEP also uses data from other sources to supply needed contextual information. For example, per capita energy usage data are obtained from the Public Service Commission and Average Daily Vehicle Miles Traveled data are obtained from the Department of Transportation. These data points are critical to further understanding factors impacting the environment. For example, as citizens demand more energy and travel more miles in their automobiles, there is an increase in the amount of pollution emissions from power generation plants and from automobiles. Both of these increases in pollution may be in 100% compliance with environmental regulations and permits.

Each quarter, the data contained in the four tiers of information in the SQPR are analyzed and used to make management decisions by the identification of "Good", "Watch" and "Focus" areas.

• "Good" areas are those in which an analysis of the tiered data indicates healthy or improving environmental conditions and high compliance rates. "Good" areas are distinguished by such characteristics as good air or water quality in Tier 1, high on-site inspection or monitoring compliance rates in Tier 2, and an appropriate number of inspections to verify compliance in Tier 3.
• "Watch" areas are those in which the data show a moderate cause for concern. For example, the compliance rate for regulatory standards in a particular district may be lower than the statewide average or compliance rates may be low in a district but only minimal formal enforcement has been taken. Such situations suggest the presence of an emerging trend or pattern and require further investigation prior to taking specific action.
• "Focus" areas are those that need to be closely monitored due to concern about persistently low compliance rates or deteriorating environmental conditions. For example, if compliance rates are persistently low despite high enforcement, the agency may consider compliance assistance alternatives or implementation of best management practices. In "Focus" areas, it is essential that management have the flexibility and support to shift resources where they are most needed to resolve problems.

F. Reporting to customers and stakeholders

Once the "Good", "Watch" and "Focus" areas have been identified, the FDEP issues Press Releases – which detail the Department’s findings. The press releases have significantly improved the FDEP’s relationship with the media and environmental groups. In conjunction with the press releases, over 1,500 copies of the SQPR are printed and distributed to the general public as well as placed on the FDEP’s Internet site. For improved accountability to stakeholders, the FDEP’s data systems are working toward an integrated facility based database system.

Transparent accountability is part of their daily work.

G. Linkages between strategic planning and resources allocations decision making, etc.

The "accountability" factor is not popular and is being addressed via the cultural change management mentioned above. However, FDEP has the strong support of the Secretary and the Deputy Secretary. The "Good", "Watch" and "Focus" areas mentioned above tie performance measurement to performance management and strategic planning since resources are redirected to address problems identified by "Watch" and "Focus" areas.

H. Benchmarking

The FDEP began their performance measurement work prior to most other environmental agencies and is viewed as the leader in performance measurement for environmental agencies. As such, very little benchmarking has been conducted. The FDEP has worked closely with the Environmental Protection Agency’s Office of Enforcement and Compliance Assurance during the development of their National Performance Measurement Strategy.

Lessons learned and other

The FDEP believes that the best method for developing performance measures is to focus on the results that you are striving to achieve (for them it is the protection of the environment/public health) then identify the contextual information needed to analyze changes in the observed results. As the information is collected, it is vital that the information be analyzed, used to make resource targeting decisions and distributed to stakeholders.