I. History/Context

The National Weather Service (NWS) is one of the five Line Offices within NOAA, which comprises over half of the budget of the U.S. Department of Commerce. The NWS has approximately 5,200 employees nationwide, and its FY 1997 budget is approximately $638 million. Even prior to the Government Performance and Results Act (GPRA), the NWS had done extensive strategic planning as part of its weather service modernization program, and explored innovative approaches to guide and measure organizational performance in a continuing way. There are several reasons why the NWS has done this.

First, NWS has always been a customer-oriented government agency focused on service to the public. NWS delivers a large number of products (weather forecasts, warnings, and advisories) every day that are used by virtually every American at least in some way.

Second, as a professional science-based agency operating 24 hours a day, 365 days a year, strategic planning and verification of organizational performance is an integral part of the way that NWS does business -- and has been doing business for a number of years. Accordingly, strategic planning and performance measurement has been institutionalized in NWS operations to maintain quality control to a degree that is probably unusual among federal agencies.

Third, the ten-year, $4.5 billion modernization plan for the NWS during the 1990's was driven in large measure by an objective determination that the performance of NWS using the science and technology of the 1980's was clearly deficient in terms of fulfilling public needs for accurate and timely weather warnings. For example, until the 1990's, the limitations of NWS's operational capabilities were such that some one third of the severe storms in the country were not accompanied by an NWS warning for the protection of the public. Moreover, approximately two thirds of the NWS's severe storm warnings that were issued turned out to be "false alarms", thereby eroding public willingness to take protective action in the face of a warning. In other words, strategic planning and performance management drove the NWS modernization, beginning as early as 1979.

II. Strategic Planning Process

While strategic planning for the modernization of NWS commenced as early as 1979, the actual budgeting for the modernization itself, as well as a full-scale NWS-wide strategic planning effort took form in the mid-1980's. An official NWS Strategic Plan, which included all of the elements of the NWS modernization was completed in 1987, and this strategic plan has provided the basis for the implementation of the modernization as well as the associated annual budgets and operating plans.

This existing strategic plan also provided the basis for a continuation of NWS and NOAA strategic planning for the 1990's. In 1990-1992, many of the elements of the earlier NWS Strategic Plan were incorporated into a broader "Draft" NOAA Strategic Plan, which was reviewed and updated throughout the NWS and NOAA leadership.

After the transition to the new Administration, a new NOAA-wide planning effort commenced in late 1992/early 1993, based upon a matrix-style management approach; and again, the NWS modernization elements, with their performance management approach, were incorporated in the new NOAA Strategic Plan -- A Vision for 2005. Also, because of the matrix-style management approach, the NWS elements were integrated with other Line Office functions of NOAA, as part of the NOAA Environmental Assessment and Prediction Mission. (The NOAA Strategic Plan contains two principal missions --environmental assessment and prediction, and environmental stewardship.)

In the new NOAA Strategic Plan, the environmental assessment and prediction mission includes the following goals:

• Advance Short-Term Warning and Forecast Services;

• Implement Seasonal to Interannual Climate Forecasts;

• Predict and Assess Decadal to Centennial Change; and

• Promote Safe Navigation.

While the NWS focused principally on the first three of the above goals, there was an effort, through the establishment of strategic planning teams for each of the seven NOAA goals, to carefully integrate each of the goals into a NOAA-wide planning effort. This process is still underway. Each of the strategic planning teams includes a mix of NOAA Line Office and Staff Office representatives who articulate the various NOAA interests for each of the objectives within that particular goal. Each team is also required to develop and update each year a 5-Year Implementation Plan for their goal. This Implementation Plan includes budget scenarios for each fiscal year, and is used as a starting point for the next fiscal year budget formulation process. In this way, the NOAA Strategic Plan is intended to drive the budget process. The Implementation Plan also contains performance measures and milestones associated with each goal.

In addition, NOAA uses an Annual Operating Plan which also contains the performance measures and milestones. As part of a Quarterly Review process, Line Offices report on progress in meeting milestones and moving toward accomplishment of outcome measures. The Strategic Plan Team Leads are asked to assess the performance of the Line Offices in accomplishing proposed work. (See Discussion in Part 2.)

III. Strategic Plan Summary

NOAA's vision for 2005 for the "Advance Short-Term Warning and Forecast Services" Goal, which is the principal focus of this case study, is to provide significantly improved short-term warning and forecast products and services for a broad spectrum of environmental events, including tornadoes, hurricanes, floods, tsunamis, and geomagnetic storms, to reduce the catastrophic impacts of natural disasters, and improve the public safety and the economic productivity of the Nation. NOAA's plan states that it will enhance its ability to observe, understand, and model the environment, and effectively disseminate products and services to users.

NOAA's plan also states that NOAA will achieve this goal by meeting the following objectives:

• maintaining modernization of the NWS;

• maintaining satellite coverage of the Nation;

• enhancing environmental observations and prediction services; and

• improving customer service through effective communication and utilization of critical weather data and information necessary for the protection of life and property.

The NOAA Strategic Plan stresses the benefits to be obtained by the implementation of this particular goal. Increasing our understanding of the environment through research and investing in new technologies will provide more accurate and timely warnings and forecasts required by the Nation. Improved forecasts will support management of water resources, and help avoid flood damage. Extended forecasts of solar and geomagnetic disturbances will increase efficiencies for space operations, and power generation and satellite communications networks. Advanced modeling techniques and more complete observations will reduce uncertainties in hurricane track prediction, saving millions of dollars through evacuation costs avoided. Accurate outlooks of future conditions will provide better information for planning weather sensitive activities over land and the oceans.

Improvements associated with the NWS modernization will more than pay for themselves. A National Institute of Standards and Technology cost-benefit analysis for the modernized NWS estimates that economic benefits to the Nation will be about 8 times greater than the costs involved. Once modernization is completed, the Nation should realize annual benefits beginning at $7 billion.

IV. Use and Impact of Strategic Planning

The NOAA Strategic Plan for the NWS has been the vehicle to design, seek approval, and implement a $4.5 billion modernization effort which includes a fundamental consolidation of NWS facilities nationwide. When the modernization process is completed, this consolidation will result in more than 50% reduction in the number of facilities (from 300 to approximately 119 Weather Forecast Offices and 13 River Forecast Centers) and a totally revamped workforce profile (professionalization of the workforce). In the current NOAA Strategic Plan, the NWS modernization is principally contained in the "Advance Short-term Warning and Forecast Services" Goal, and comprised about 59% of the total NOAA FY 1997 appropriation.

V. Costs

Numerous NWS and NOAA personnel now conduct strategic planning activities as a vital component of their normal work responsibilities. There has been much reliance on the existing planning and budget staffs, as well as programmatic staff and policy officials and the matrix style management process within NOAA to develop and implement this strategic plan. This modest investment is in stark contrast to the actual costs of the $4.5 billion modernization effort, which is leading to a dramatic improvement of forecast and warning services to the public.

VI. Lessons Learned

NOAA is well underway toward completing the largest modernization and restructuring effort in its history to improve the severe weather, flood warning and forecast services in the U.S. Major technological advances in meteorology and hydrology, coupled with the need to replace obsolete and increasingly unreliable equipment provide an unprecedented opportunity to improve weather and hydrologic services. These replacement systems, along with observations from advanced satellites, and newly developed storm-scale forecasting techniques, will continue to improve the timeliness and accuracy of severe weather and flood warnings to the U.S. public.

The NOAA Strategic Plan for the NWS became the vehicle for the systematic fleshing out of goals for modernization of the NWS, including competing goals between and within the Executive Branch and the Congress. While the implementation of new technology was the driving force at the outset, during the course of the strategic planning effort, levels of service, quality and accuracy also became focal points. The development of the goals of the plan forced an honest policy debate about ways to implement new technology, including major restructuring to take advantage of the efficiencies available, and also ways to improve the level and quality of services to the public.

VII. Next Steps

The modernization and restructuring of the NWS is about two-thirds completed. In addition to continued monitoring, updating, and evaluation of the modernization occuring within the Executive Branch, the Congress is also involved in overseeing the final implementation of the plan over the next 5 years. Under P.L. 100-685, and an amended version, P.L. 102-567, Congress recognized, and in effect, adopted the NOAA Strategic Plan for the NWS, but also established a certification process for the closure of weather service facilities. For each closure of a facility, Congress required a certification by the Secretary of Commerce that no degradation of services to the public would result for that particular area. Although NOAA has characterized this process as being too burdensome, and has proposed legislation to streamline the process, the mechanism has assisted in "legitimizing" the closure of outdated weather service facilities by allowing for greater public and Congressional participation.

Maintaining NWS Modernization is only one of the objectives of the "Advance Short-Term Warning and Forecast Services" Goal of the NOAA Strategic Plan -- A Vision for 2005. The other objectives are as follows:

• Enhance observations and predictions: The NWS Modernization represents only the first step toward providing an overall system of improved environmental forecasts. Additional service improvements through the U.S. Weather Research Program, the National Space Weather Program, and the Coastal Forecasting System can only occur through well organized collaborative research efforts teaming federal and university organizations focused on a combination of scientific, technological, and programmatic advances. This will include research in the areas of predicting weather, oceanic conditions, and the space environment. New observing systems will be developed, tested, and implemented to replace aging systems and to fill large data voids crucial to improving warnings and forecasts. As we head into the 21st century, benefits and service improvements in very short range forecasts of flash flood, severe thunderstorm, and tornado warnings as well as service improvements in forecasts of winter storms, geomagnetic storms and coastal forecasting will be realized through enhancing observations and predictions.

• Maintain satellite continuity: NOAA is responsible for the procurement, launch maintenance, and operation of a system of polar-orbiting and geostationary satellites to provide near-continuous observations of the U.S., adjacent oceans and space environment. These observations, and observations accessed from other U.S. and foreign satellites, provide data to monitor the Earth's surface and space environment conditions for a wide range of federal agencies, state and local governments and private users. The collection, processing, and analysis of the satellite data, is primarily used for forecasts and warnings. Improvements in temperature, snow, aviation, and precipitation forecasts, as well as severe coastal event warnings are directly related to maintaining satellite continuity.

• Improve service communication and utilization: NOAA's short-range prediction improvements must be delivered to, and utilized effectively by, communities and groups at risk in order to ensure the protection of life and property from natural hazards. This requires assessing the customer's critical information needs, meeting those needs through new products and services, educating customers on products and services, effectively communicating critical information, taking full advantage of technological advances in the information processing and communications revolution, and involving the customer in the assessment of the effectiveness of the resulting new products and services. NOAA will expand customer interaction through the use of appropriate methodologies and will provide resources to ensure maximum customer involvement. This is imperative to translate scientific improvements into natural disaster reduction for the Nation.

In addition, NOAA will be actively moving ahead with two other separate, but related goals of its strategic plan which include:

• Implement Seasonal to Interannual Climate Forecasts: NOAA, working with academic and multinational partners, will provide one-year lead-time forecasts of known skill of global climate variability, especially El-Nino and the consequent precipitation and surface temperature distributions. These forecasts will increase society's ability to mitigate economic losses and social disruption.

• Predict and Assess Decadal to Centennial Change: NOAA will provide science-based options for decisions regarding decadal-to-centennial changes in the global environment, specifically for: climate change and greenhouse warming; ozone layer depletion; and air quality improvement.

Part 2

DEVELOPMENT AND USE OF OUTCOME INFORMATION
IN NOAA'S NATIONAL WEATHER SERVICE

With the completion of the NWS Strategic Plan in 1987, which would totally modernize and recast the way in which the NWS did business, there was also an effort to develop performance measures which could address progress toward modernization and measure the actual impact of new prediction and forecast capabilities. While these performance measures have evolved during the modernization process, there has been a continuing effort to make them outcome-based and focused on the public's needs for prediction and forecast services.

II. Development of Indicators

Performance measurement in the NWS takes several forms at the present time. Measures range from relatively routine quantitative parameters of operational readiness to ad hoc assessments of NWS's actual conduct in specific severe weather events, such as "lead time" and % accuracy for tornado, severe thunderstorm, and flash flood warnings. NWS is also attempting to move beyond these traditional performance measures in order to achieve a higher degree of performance in terms of important national goals where weather is a factor. Each type of performance measurement will be reviewed briefly.

• The first type of performance measure is the routine statistic related to some aspect of organizational activity: Typically, such a measure would assess the operational readiness of some important piece of equipment or technological system, such as NEXRAD radars or AWIPS. For example, each weather radar normally operates at least 95% of the time. Some high degree of operational reliability is the goal of performance management in this case. Although a weather radar operating normally is one prerequisite for a "good" weather forecast, this measure alone says nothing about how "good" a forecast actually is.

• The second type of performance measure focuses on the "skill" or accuracy of a weather forecast product in objective, quantitative terms: Typically this measure is for an individual forecaster or for a run of a particular computer forecast model, and the "verification" of the forecast parameter, such as temperature, is obtained by comparing it with the actual measurement of that parameter. This measure seeks to score the accuracy of a weather forecast, but it says nothing about the utility of the forecast to some user even if it is accurate.

• The third type of performance measure is directed at assessing how well the NWS performed in providing adequate public warnings in specific cases of severe weather: Official NWS "disaster surveys" are undertaken when the loss of life or property exceeds a certain threshold, and extensive investigations and interviews are conducted with parties external to NWS, such as State emergency management officers, the objective is to ascertain how well the NWS forecast and warning services performed in actually saving lives and reducing property losses in severe storms such as tornado outbreaks.

Unlike the first two types of performance measures which target objective, quantitative measures of internal NWS performance, the NWS disaster survey seeks to determine the external impact of NWS forecasts and warnings to protect life and property. In a typical year, no more than several NWS disaster surveys are implemented and the primary purpose is to identify any mistakes that were made, any breakdowns in communication that occurred or generally what lessons could be learned for future action.

III. Indicators of Outcome/Results

For the most part, the performance measures collected by the NWS are quantitative, and the NWS has considerable experience in collecting and verifying the data to support them. (See Attachment 1). These are largely "lead-time" and accuracy indicators which provide a very quantitative, measurable, and reliable benchmark -- and in most cases, a benchmark which is outcome-based.

Building upon the conceptual approach of the NWS disaster survey which focuses on the external impact of NWS forecasts and warnings, there are opportunities to extend NWS performance measures to more than a few ad hoc cases and to the socioeconomic utility of weather information that goes beyond cases of severe weather. The overriding goal will be to seek higher levels of NWS performance in satisfying national needs where weather is a factor.

IV. Use and Impact of Information on Outcomes/Results

The performance data currently collected by the NWS has been collected and verified for a number of years. Since the GPRA, there have been further efforts to refine and validate the existing performance measures being used by the NWS, but because of the earlier strategic planning process, most of the measures remain valid and provide a fairly reliable measure of accountability. The performance measures (see Attachment A) are the basis for the collection of data by region and by Weather Service Forecast Office, and can therefore provide a basis for performance and accountability at those levels. Of course, inherent geographical variations exist in the "difficulty of forecasting" weather and hydrologic events, and these are taken into account by NWS and NOAA leadership in their reliance on these tools.

These performance measures, which are for the most part quantitative, have been fairly reliable instruments to evaluate the effects of the NWS modernization. For example, modernization efforts to date already have demonstrated changes in warning lead times for severe weather events from less than zero to 10 to 15 minutes. Final improvements in the accuracy and timeliness of these weather and flood warning and forecasts will result in saved lives and reductions in losses from weather disasters. These improvements, which are already being borne out by the performance measures, will also result in very large economic gains for highly weather dependent sectors of the U.S. economy such as transportation and agriculture.

V. Costs

Although few NOAA and NWS staff are dedicated solely to these efforts, there is strong reliance on the existing planning and budget staffs, as well as programmatic staff to develop, validate, and verify existing performance measures. It is estimated, in the aggregate, that the NWS commits approximately $1.5 million and 20 to 30 FTE's per year to the task of compiling and verifying performance measurement data. This modest investment is in stark contrast to the actual costs of the $4.5 billion modernization effort, which is leading to a dramatic improvement in forecast and warning services to the public.

VI. Lessons Learned

The lessons are still being learned as the NWS proceeds through its process of modernization and continues to validate and verify its performance measures, while continually improving its short-term forecasting and prediction capabilities. Also, the NWS is evaluating additional areas of performance related to its many customers and potential new partnerships with the users of NWS data.

The NWS has benefitted greatly from its strategic plan developed at the outset of the NWS modernization. The plan simplified the performance measurement process -- because everyone was aware of the goals they were trying to achieve. The performance measures are a logical outgrowth of the strategic plan, and the results of the modernization are already measurable. For example, the use of new Doppler radars has resulted in positive "lead-times" for the first time ever in the case of some severe weather events.

VII. Next Steps

Several principles and concepts offer promise for a more sophisticated approach for NWS to manage and measure its performance as an agency to serve the public. These principles and concepts include the following:

• Focus on customers: Although every American can be considered a NWS customer, there exists a broad range of "aggregations" of NWS customers that can greatly simplify transaction costs to define customer needs and guage NWS fulfillment of those needs. Fundamental customer relationships for the NWS include State Governors and various State agencies that have some connection to weather. One way to ascertain the "agenda" of needs and opportunities for a particular state is to ask the State Government.

• Emphasize external impact: Most governmental organizations prefer to be evaluated in terms of those performance measures for which the organization has a relatively high degree of internal control over the successful accomplishment of planned goals. However, the true "value" of services of a governmental agency such as NWS lies in their practical application by external parties. Although NWS has less control over successful organizational performance where extraneous factors contribute to success, the socioeconomic benefits of an enhanced focus on external impacts involving other parties is potentially very high.

For example, if more Americans die in highway accidents when weather is a factor than for any other weather related risk, the NOAA could work with a State Governor, the U.S. Department of Transportation and the American Automobile Association to seek a reduction in these highway fatalities through public education and other steps beyond issuing accurate NWS forecasts.

• View service delivery as problem solving: One way to ensure a greater focus on external impact of an agency's programs is to change the perspective of service delivery. This perspective could be changed from the passive issuance of high quality information products, such as accurate weather forecasts, to a view that the societal benefits of a publicly supported weather service can only be maximized if the NWS takes the initiative to work with external parties to help "solve" urgent problems on their "agenda" where weather information can be useful. Strategic partnering with federal and State agencies in joint problem solving is one approach to achieve an effective problem solving posture.

• Adopt an experimental, learning approach to performance management: In contrast to traditional control-oriented performance measures such as the Management by Objectives (MBO) type of approach, achieving and measuring a higher degree of fulfillment of societal goals will require a more pragmatic and flexible NWS approach that stresses and rewards innovation. For example, in the case mentioned above involving the reduction of highway fatalities in adverse weather conditions, fostering of behavioral changes by drivers, such as reducing vehicle speed in poor visibility, could be pursued by a coalition of State highway officials, NOAA and private organizations in a manner parallel to the anti-drunk driving campaign of the Mothers Against Drunk Driving. In this case, would a reasonable annual reduction of highway fatalities in a particular state for performance management purposes be 5%, 10%, or 25%? The point is -- any reduction is worth pursuing as a matter of public policy, and an experimental approach could evaluate various strategies in pragmatic ways. This could be an approach adopted by NOAA and the NWS for performance management in the future. Such an experimental approach would include innovative partnerships with federal and state agencies as well as the private sector.

It should be stressed that outyear performance measures are dependent on a stable funding profile and take into account improved use of the Weather Surveillance Radar (WSR-88D), new satellites, improved forecast models, new and continued research of the U.S. Weather Research Program, investments in critical observing systems, Coastal Forecast System activities, and the implementation of the Automated Weather Interactive Processing System (AWIPS).

As the National Weather Service continues to change, upgrade its capabilities, and reorganize, so too are the various sectors of the economy. To keep pace with changing demands and to assist in its strategic planning process, NOAA has contracted The Kanawha Institute for the Study of the Future to conduct a study of the present and future needs for weather data and information by business and industry. The societal and economic trends identified in this study will enable NOAA to re-evaluate its current strategic planning goals and objectives, and continually assess its performance measurement process.

ATTACHMENT 1

KEY FY 1998 PERFORMANCE MEASURES
National Oceanic and Atmospheric Administration

This measures the lead time (in minutes) and accuracy (in percent) for successfully forecasting a Flash Flood Warning. A Flash Flood Warning is issued when flooding has been reported or is imminent. Flash flooding is the #1 weather-related killer in the United States, with nearly 140 fatalities each year. Advance warning of flash floods will enable people to take steps toward protecting life and property.

This measures the lead time (in minutes) and accuracy (in percent)for successfully forecasting a Severe Thunderstorm Warning. A SevereThunderstorm Warning is issued when thunderstorms produce hail at least 3/4 inch in diameter, winds of 58 mph or higher, or tornadoes. Of the estimated 100,000 thunderstorms that occur each year in the United States, about 10 percent are classified as severe (with its associated lightning, flash floods, straight-lined winds, large hail, and possible tornadoes). Advance warning of these severe storms will enable people to take steps toward the protection of life and property.

This measures the lead time (in minutes) and accuracy (in percent) for successfully forecasting a Tornado Warning. A Tornado Warning is issued when a tornado has been sighted or indicated by radar. Tornadoes are the most destructive storms in the United States, with over 800 tornadoes, 80 deaths, and over a thousand injuries reported each year. Advance warning of these violent storms will enable people to take steps toward the protection of life and property.

This measures the lead time (days in advance) for successfully forecasting one inch of precipitation as compared to a 1971 precipitation forecast. The NWS is now successfully forecasting one inch of precipitation 2.3 days in advance as compared to 1.0 day in advance in 1971. In light of the devastating socio-economic impacts of flooding and significant snowfalls, quantitative precipitation forecasts (QPFs) have emerged as a critical facet of the end-to-end forecast process. Improved and early lead time of one inch of precipitation or more will promote timely and accurate issuance of flood and winter storm forecasts which are essential for the preservation of life and property.

This measures the accuracy of a landfalling hurricane (in kilometers) with a 24-hour lead time. Hurricane Warnings are issued when hurricane conditions are expected within 24 hours. Hurricanes are powerful, tightly coiled weather systems with high winds, heavy rains, tornadoes, and a battering storm surge along the coastline near where a hurricane makes landfall. Earlier and improved predictions of storm track and intensity will enable people to take steps toward the protection of life and property, as well as reduce the size of the warning area.