Despite a growing number of efforts to measure and report on health care quality, useful information is neither uniformly nor widely available. Improving our ability to measure quality has been the object of significant public and private sector activity over the last decade, reflecting the expectation that measurement can serve both as a catalyst and a tool for improvement. While considerable advancements have been made in the quality measurement field in recent years, current efforts fall short of fully meeting users' needs, and often are duplicative and unduly burdensome on health care providers, plans, and others.
The Commission has identified several steps critical to advancement. First, core sets of standardized quality measures should be identified to address the common information needs of individual consumers, group purchasers, health plans and providers, oversight organizations, and public health and policy officials. These measures must focus on areas that can have a meaningful impact on the quality of health care and be consistent with national aims for improvement. In addition, new and better quality measures should be developed to fill important gaps in existing measurement sets, and steps should be taken to ensure that comparative information on health care quality is valid, reliable, meaningful, comprehensible, and widely available through multiple means of dissemination.
A framework and capacity for quality measurement and reporting should be developed in tandem with the standardization of quality measures for reporting. Specific functions to be undertaken include the following:
All sectors of the health care industry should support the focused development of quality measures that enhance and improve the ability to evaluate and improve health care. Comprehensive sets of quality measures are needed to reflect the full continuum of health care, but measures currently are lacking in a number of important areas. Types of measures that are needed include the following:
Quality measurement requirements and the detailed specifications for measurement and reporting should be determined through a stable and predictable mechanism. The Commission's specific recommendations on the creation of such a stable and predictable mechanism -- a Forum for Health Care Quality Measurement and Reporting -- are described in Chapter 5.
This entity, which would be responsible for designating core sets of measures for reporting for each sector of the health care industry, should:
Steps should be taken to ensure that comparative information on health care quality is valid, reliable, comprehensible, and widely available in the public domain. Specifically:
For quality measures to play such a role in health care decisionmaking, comparative information generated through measurement must be available to those who face choices among alternative health care providers or organizations. One group of potential users of this type of information is individual consumers, who usually have at least some choice of primary care practitioner and other providers within their health plan, and who sometimes also have a choice of plans. Public and private sector group purchasers also face choices among potential contractors, normally health plans but sometimes also providers. Finally, health plans themselves can use quality measures in their decisionmaking by using them to select providers for participation in their networks.
Quality measurement to support oversight efforts. Public and private oversight organizations utilize quality measurement in their efforts to safeguard the public from poor-quality health care, distinguish those who furnish better care, and stimulate ongoing quality improvement efforts. Various entities have been established to license, accredit, or certify health care providers and organizations (see Chapter 9). Quality measurement provides these entities with information to feed into their determinations as to whether a provider or organization can serve a defined market, whether financial or other sanctions should be imposed, or whether an entity should be rewarded in some manner for providing exemplary care.
Quality measurement for public health initiatives and policymaking. Quality measures can be used to support public health initiatives and public policymaking, providing a means of evaluating the quality of care provided through alternative financing and delivery systems, in particular geographic areas, and to vulnerable groups within the population. Public health planners can use information from quality measures to examine the need for educational campaigns or other initiatives in specific areas and to gauge the subsequent effects of those initiatives. Public policymakers can use measures to examine aspects of health care quality that might be affected by changes in policies, like those that dictate methods of paying providers that serve beneficiaries of public programs.
Role of quality measures in internal quality improvement initiatives. Quality measures also can provide the tools that allow health care providers and organizations to undertake efforts to improve the quality of care they furnish. They thereby provide a means of influencing or responding to the decisions of purchasers and oversight entities, as well as any actions in the public health and public policy arenas. The measures can serve in focused improvement programs, whereby quality is measured both before and after an intervention designed specifically to improve the way in which care is provided or its outcome.
The work of these groups has been supplemented by additional quality measures and sets of measures developed by numerous professional associations, such as the American Nurses Association (ANA, 1995); organized health systems and providers, including both health plans and hospitals; and research and consulting organizations. Development of new measures and measurement initiatives is continuously being undertaken to fill perceived gaps in the existing measurement sets, whether in the areas of health care addressed (e.g., treatment of chronic conditions, preventive care), the units of analysis (e.g., health plan, hospital, medical group), the user orientation (e.g., providers, consumers, purchasers), or the populations of interest (e.g., commercial health plan enrollees, Medicare beneficiaries, Medicaid beneficiaries).
This continuing growth has been so substantial that efforts are under way to create electronic databases to catalog and describe the variety of measures. One of these, an AHCPR-sponsored project known as CONQUEST, has identified more than 53 separate measurement sets containing more than 1,100 clinical quality measures. Separate projects have also been undertaken to create an inventory of population and health systems performance measures.
Measures of health care quality are being used by a growing number of private and public sector organizations at the national, State, and local levels that have undertaken initiatives to collect and disseminate information on quality. These groups include the Maryland Health Care Access and Cost Commission, which has used data from enrollee and practitioner surveys and from health plans to generate comparative information on performance; the Minnesota Health Data Institute, a not-for-profit entity that distributed comparative performance reports on health plans; the Alabama Health Care Council, an employer purchasing coalition that collected and analyzed outcome and cost information on hospitals; and the Cleveland Health Quality Choice project, a coalition of employers and providers that examined the technical quality of care and patient satisfaction with local hospitals. JCAHO and NCQA have developed national databases of information on the performance of institutional providers and health plans across sets of quality measures. The American Medical Association also is taking steps to establish a national database of information on the performance of individual practitioners.
Nor do current measurement initiatives serve public health and quality improvement objectives as well as they might. Incentives to improve quality have been diluted by measurement efforts that vary widely in their aims and scope, and that have been, at best, only informally coordinated. A lack of clearly articulated national priorities for improving health care quality has meant that the superior ability to measure quality in specific areas or along certain dimensions may have diverted attention from issues of greater importance with respect to quality.
Furthermore, the lack of widely agreed-upon priorities and standards for quality measurement has been a source of frustration and inefficiency. Most health plans and providers now produce data in response to multiple, customized requests from purchasers, oversight bodies, consumer groups, and others. For example, a health plan might be required to calculate and report on how it cares for individuals with diabetes in several different ways to respond to uncoordinated demands. Similarly, a health care practitioner who participates in several health plans may need to fulfill various requests for data of different types and formulations. The resulting information is not always comparable, meaningful, valid, or reliable, and is disseminated in different ways and through various means.
To increase the effectiveness and efficiency of measurement, core sets of quality measures for each sector of the health care industry should be identified for reporting in a standardized way. Measurement sets should be designed to meet the information needs of potential audiences and should reflect defined national aims for improvement in health care (see Chapter 3). In this way, their development would ensure the availability of comparable information on quality and increase the potential impact of measurement while alleviating the burden of complying with reporting requests. Developing standards for measuring and reporting on a core set of indicators also would allow health care providers and organizations to reallocate resources to focused quality improvement and to the generation of additional information, as needed, to supplement the core measures. Finally, core measurement sets would assist in tracking progress in addressing high-priority areas established as national aims for quality improvement.
Create synergy of improvement efforts throughout the health system. Core sets of health care quality measures should help to create synergy of quality improvement efforts undertaken throughout the health system. While individual measures and measurement methods specified for each sector (e.g., health plans, hospitals, long-term care facilities) will vary to account for differences in the provision of care and in the health data and information systems available, common measures and areas of focus should be used where possible. Common measures will need to be used across delivery systems, including both fee-for-service and managed-care models. Use of consistent measures over time will be needed to foster the ability to track trends, establish targets for quality improvement, and assess the levels of increases in quality.
Capture multiple dimensions of quality. To provide a comprehensive and balanced assessment of health care quality, measurement efforts will need to capture multiple dimensions of quality. These dimensions include the structures of a health care organization or system of care that can influence quality, the processes associated with the delivery of care, and the outcomes obtained (Donabedian, 1966). Another framework for classifying dimensions of quality delineates three dimensions of quality:
These dimensions might be modified or enhanced to reflect the expected uses of the information generated through quality measurement and reporting efforts. For example, value might be included as an additional aspect of quality relevant to individual consumers and purchasers who would use the information in health care decisionmaking. Furthermore, the dimension of acceptability might be broadened to include the perspectives of health care practitioners so as to obtain a richer view of health care quality.
Serve the needs of consumers. Quality measurement should focus on serving the needs of consumers, who can use information on quality to make choices among health care professionals, institutional providers, health plans, and systems of care. Orientation toward consumers does not preclude addressing the needs of other users of information on quality. A sound measurement strategy must take into account the interests of health care providers, purchasers, plans and other health organizations, oversight groups, policymakers, public health representatives, and others whose decisions directly affect the well-being of patients.
Address quality of care for vulnerable groups. Quality measurement should account for concerns about vulnerable groups within the population (see Chapter 8). There is a clear need to increase the level of attention paid to these groups, including both those who, because of chronic illness or disability, have many interactions with the health system, and those who have difficulty accessing the system and may be most likely to fall through the cracks during this period of rapid health system change. In addition, there is a particular need to pay attention to the measurement of quality for children's health care.
An increased focus on vulnerable groups could be accomplished through two measurement strategies. First, measures focused on specific quality issues that disproportionately affect vulnerable groups (e.g., measures of the quality of chronic care) could be included in reporting sets. Second, separate analyses of the quality of care provided to vulnerable groups could be conducted and reported using the full reporting sets, to the extent feasible in light of methodological issues and cost constraints.
Encourage innovation. Identification of core sets of quality measures is not intended to lock in approaches to quality measurement and reporting. Innovation can be fostered by calling for the development of new measures to augment, complement, and enhance the required measures; updating core measurement sets periodically to reflect advances in measurement techniques; and placing limits on the breadth of standardization efforts (e.g., number of measures) to ensure that compliance with the core reporting sets does not consume resources better devoted to innovation or internal quality improvement.
Provide efficient mechanisms for data reporting and collection. A centralized data repository may need to be created to facilitate the reporting of core information on quality and the dissemination of such information to the public. Use of computer technology -- particularly the Internet -- could allow for such a repository to be affordable and usable by both the reporters and the users of the information.
Evaluate measures and measurement sets. Measures need to be evaluated, both individually and as sets of measures. An important part of the process will be to agree upon the criteria used to evaluate individual measures and measurement sets. Examples of criteria that might be identified for evaluating individual measures include the following:
Somewhat different criteria would be established for the evaluation of measurement sets. For instance, various criteria might be identified to assess whether a potential set was sufficiently comprehensive. Does the set address the full spectrum of health care? Does it incorporate measures of multiple dimensions of quality (e.g., technical quality, accessibility, acceptability)? Does it include various types of measures (e.g., structure, process, outcome)? The representativeness of measurement sets might also be of interest, given that an undue concentration of measures in a specific clinical area could provide a distorted view of performance. Another key evaluation criterion might be the measurement burden associated with a set. Is the set as concise as possible? Is it redundant? Can measurement be conducted efficiently, with a minimal burden on providers and health care organizations?
Promulgate measures and specifications for measurement and reporting. Measurement sets will need to be formulated, evaluated, and promulgated following a field trial or test period as needed for validation. Measurement specifications need to be developed that address source data sets and data elements to be used in making measurements; the measurement methods, including risk adjustment formulas, to be used; instructions for an independent, external audit prior to public reporting; and reporting formats and communications modes to be used. Finally, measures will need to be reassessed subsequent to their promulgation to determine their utility in practice.
Another issue to be addressed is the question of how best to promote reporting of clinical errors or adverse outcomes (see Chapter 10). Measurement of clinical errors or adverse outcomes can offer significant opportunities to improve health care, but public reporting of health care quality problems is likely to be problematic. Public reporting of such information could lead to underreporting of problems by health care providers who are concerned about potential legal liability or affected by social, cultural, and financial barriers to acknowledging mistakes. Underreporting would diminish opportunities to identify areas in which quality improvement is needed and would yield inferior information to use in health care decisionmaking.
A number of alternatives for addressing this issue could be considered. Legal protection from liability claims associated with reporting of adverse events also should be considered to foster complete and accurate reporting. Further analyses are needed to identify how changes in medical liability and quality oversight systems might affect both the likelihood and the accurate reporting of adverse events. An alternative strategy would be to distinguish between measures to be used for internal quality improvement and those to be used for external reporting. Finally, steps could be taken to ensure that publicly released information was aggregated in such a way as to render it not individually identifiable.
Several aspects of health care are ripe for the development and testing of new and better quality measures. For instance, many more measures of preventive and acute care have been established, as compared with measures of chronic conditions. The need for measures of health care outcomes, including functional status, also has been identified as an important area to be addressed. In terms of specific clinical areas, mental health and substance abuse care, pediatric care, care for specific disabilities, geriatric care, and care for injuries or other traumas have all been identified as gaps. Better measures of the interpersonal aspects of health care that influence patients' perceptions of quality are also needed. Finally, as discussed further in Chapter 8, there is a need for measures designed to assess the health care procured by vulnerable populations. Quality measures are lacking in these areas for a variety of reasons, including problems with the available data and data systems (see Chapter 14), lack of accepted clinical practice guidelines, and an insufficient investment of resources. Fully addressing all aspects of health care in quality measurement sets will be not be possible, but a wider variety of measures are needed to allow for more comprehensive assessments.
Existing measures also must be adapted to a broader scope of use. For example, many quality measures were developed for use in managed care plans, which serve a defined population that is relatively large in comparison with the number of patients served by an individual physician or medical group. To meet measurement needs imposed by changes in the financing and delivery of health care, measures must now be adapted or newly developed to measure the quality of care provided by individual practitioners and obtained in fee-for-service or differently defined systems of care. In the same vein, quality measures that rely on patients' self-reports must be translated to other languages to be meaningful for the full population.
This need for additional quality measures raises concerns about the potential for information overload as well as risking an imbalance of resources allocated to measurement at the expense of improvement. Group purchasers report that the variety and amount of performance information to be processed in making purchasing decisions is a barrier to doing so effectively (Hibbard et al., 1997). Summary measures that combine information on quality across multiple dimensions could help in addressing this concern, as could the use of crosscutting measures that provide information about the quality of care obtained by patients across multiple conditions (e.g., measures of access to specialty care). A defined set of national aims for improvement and an articulated framework for measurement that takes those aims and users' information needs into account would provide the sense of priorities needed to process information adequately.
At present, there is very little consistency in how measures with the same conceptual definition are being operationally defined and technically specified (Coltin, 1997; McCormick et al., 1997). For instance, alternative quality measures might be available to address cesarean section rates that differ in terms of their operational definitions (e.g., the age range of women to be included in measurement) or technical specifications (e.g., the timeframe or type of data to be used). Health care organizations and providers today are called upon to report on pediatric immunizations in one way for a consultant acting on behalf of a large employer, another way for the State Medicaid program, a third way for the State regulators, a fourth way for accreditation purposes, and a fifth way for the local business coalition (Udvarhelyi, 1997). These variations do little to add to the overall store of information on health care quality and detract from the ability to compare results.
Assurance that health care providers and organizations were using the same types of data, including the same segments of the population, and adjusting for risk in the same ways in their calculations would allow for greater confidence in the comparability of reported information. It would also provide greater ability to trend improvement in quality over time, a capacity that is necessary for setting targets for improvement and gradually increasing the overall level of quality. Agreement on appropriate measurement specifications for conceptual measures would be valuable in reducing unnecessary duplication of effort and the wasted resources that this entails.
Other industries, as well as the health care industries of other countries, have begun to address inconsistencies in the specifications and operational definitions of data concepts by establishing data registries. These registries serve to promote consistent use of data that need to be interchanged between organizations. Examples of such efforts include the National Health Information Knowledgebase of the Australian Institute of Health and Welfare and the Basic Semantic Repository Project of the International Organization for Standardization of Data Elements.
Make appropriate adjustments for risk. Risk adjustment is sometimes needed to ensure that fair and valid comparisons are made on the basis of legitimate differences in health care quality, since characteristics of populations that are outside of the control of a health plan or provider may influence measurement results. Methods used to adjust quality information for risk should be subject to external scrutiny to ensure that they meet explicit standards of clinical credibility and statistical rigor. Evidence that the need for risk adjustment has been considered in the development of a specific measure should be examined prior to promulgating a measure for reporting. Specifications for risk adjustment should be included or not included in measurement standards as appropriate to each measure.
The objective of risk adjustment is to account for differences in the mix of patient factors across plans or providers so that comparisons of how their patients fare can be fairly made. The underlying assumption is that outcomes result from a complex mix of factors that relate, in part, to intrinsic patient characteristics. The mix of persons covered by different health plans or treated by different providers varies, reflecting the multiple factors that affect the way persons select their sources of care (e.g., the nature of their health needs, financial considerations, preferences, and expectations). These differences in mix are important for quality measurement: Persons with complex illnesses, multiple coexisting diseases, or other significant risk factors are more likely to do poorly, even with the best care, than healthy individuals (Iezzoni, 1997).
The appropriate use of risk adjustment techniques is essential because using measures that fail to account for important differences in the populations served runs the risk of penalizing in the marketplace those plans and providers that serve the sickest and neediest populations. Failure to risk-adjust before comparing performance across health plans or providers can have several consequences. Without adequate risk adjustment, resulting information could be inaccurate or misleading. Consumers, policymakers, and other health care stakeholders will not have valid information for decisionmaking. Another goal of quality measurement is to motivate improvement of care. Without risk adjustment, providers often perceive that they are treated unfairly. Providers could avoid high-risk patients who are more likely to do poorly. Furthermore, failure to adjust for risk hampers attempts to engage providers in a meaningful dialog about improving performance.
Despite the need for risk adjustment methods, current techniques inherently are imperfect and likely to remain so for several reasons: lack of requisite data, especially pertaining to important nonclinical attributes of patients (e.g., socioeconomic status, psychosocial issues, functional abilities, preferences and expectations about health care); problems with existing data (e.g., completeness and accuracy); unavailability of appropriate data sets on which to develop and validate risk adjustment methods; and, perhaps most important, the absence of well-thought-out, evidence-based conceptual models of risk factors for performance indicators of proven validity (Iezzoni, 1997).
Similarly, there are limits in what can reasonably be expected of risk adjustment. It will never be possible (or even desirable) to adjust for every characteristic of patient populations that could affect quality measurement. Gathering the necessary data is expensive and logistically complicated. Collecting some types of information may even be impossible by any practicable means. Given that science cannot guarantee that risk adjustment will be perfect, decisions about risk adjustment will inevitably engender controversy -- with legitimate arguments for and against using methods with inevitable shortcomings. Despite these limitations, the Commission recommends that development, testing, and implementation of risk adjusters proceed as rapidly as possible.
Provide for an independent, external audit. Conducting an independent audit of processes used in data collection and analysis is necessary to ensure that publicly reported information is comparable and accurate. There are numerous reasons why externally reported data may be inaccurate: measurement specifications that fail to provide detailed instructions; errors on the part of health care organizations in interpreting specifications; use of different source data sets to undertake measurement; incomplete or inaccurate source data sets; and differences in clinical coding systems (National Committee for Quality Assurance, 1995). Analytic mistakes can also lead to reporting errors. By providing assurance that information is accurate, auditing can increase the integrity and usefulness of publicly reported information on health care quality.
If undertaken by an entity that is independent and external to the organization whose performance is being examined, auditing may serve to allay concerns about the accuracy and validity of reported information on health care quality. Recent studies have suggested that one reason many large health care purchasers have refrained from making decisions based on performance data is concerns about whether they are accurate (GAO, 1997; Hibbard et al., 1997). Certification of accuracy by an entity that is known to be independent and external to the organization whose performance is being measured would provide an assurance that reported information provides a sound basis for decisions.
Independent, external audits can be resource-intensive and are not required for all uses of quality data. For instance, health care organizations might determine individually, on a case-by-case basis, whether information to be used internally for quality improvement efforts required an external check on its validity. Other types of uses of quality information, such as research uses, could rely on peer-review processes or alternative validation methods.
Establish effective approaches for dissemination and transmission of information. Comparable quality information should be placed in the public domain and made available to the full array of interested audiences. Problems attaining information known to be comparable and reliable have been among the greatest obstacles to the work of consumer advocacy organizations and others interested in assisting consumers with their health care choices (Krughoff, 1997). Routine release of this type of information -- and assurance that such information is easily accessible and affordable -- would greatly facilitate the work of employee benefits departments, professional and trade associations, national and local media, and others who would disseminate the information to those who could use it in their decisionmaking (see Chapter 7). It might also reduce the burden on health care organizations and providers of responding to a number of uncoordinated requests for information on quality.
Placing measurement results in the public domain is necessary, but not sufficient, to promote their use in health care decisionmaking. More research is needed to determine how best to make information on quality accessible and usable. To foster widespread awareness and use of the information, quality measurement findings should be reported through multiple modes and dissemination channels. Standard reporting formats and accessible language will need to be used. Further, steps may need to be taken to facilitate interpretation of the information provided, particularly for certain vulnerable groups within the population. Various types of consumer assistance programs, such as those provided through employers' benefits departments, labor unions, or independent ombuds organizations, may play a role in this effort.
An information campaign -- targeted to consumers and other information users -- will also be needed to promote and support appropriate use of information on health care quality. The availability of comparative information on quality should be advertised to reach a wide audience. Potential audiences will also need to be educated about the value of using such information in making health care decisions, the limitations of measurement, and the parameters of appropriate use. Strategies for disseminating quality information to consumers are discussed in Chapter 7.
Coltin, Kathryn, oral testimony before the Subcommittee on Quality Measurement of the Advisory Commission on Consumer Protection and Quality in the Health Care Industry, July 21, 1997.
Donabedian, Avedis, "Evaluating the Quality of Medical Care," Milbank Memorial Fund Quarterly 44:166-203, 1966.
General Accounting Office, Management Strategies Used by Large Employers to Control Costs, GAO/HEHS-97-71 (Washington, DC: May 1997).
Hibbard, Judith H., Jacquelyn J. Jewett, Mark W. Legnini, and Martin Tusler, "Choosing a Health Plan: Do Large Employers Use the Data?" Health Affairs 172-180, November/December 1997.
Iezzoni, Lisa, ed. Risk Adjustment for Measuring Healthcare Outcomes (Chicago: Health Administration Press, 1997).
Krughoff, Robert, oral testimony before the Subcommittee on Roles and Responsibilities of Group Purchasers and Quality Oversight Organizations of the Advisory Commission on Consumer Protection and Quality in the Health Care Industry, November 18, 1997.
McCormick, Kathleen, Alice L. Renner, Robert Mayes, et al., "The Federal and Private Sector Roles in the Development of Minimum Data Sets and Core Health Data Elements," Computers in Nursing 15(2)(Suppl.):S23-S32, March/April 1997.
National Committee for Quality Assurance, NCQA Consumer Information Project: Focus Group Report (Washington, DC: 1995).
Palmer, R. Heather, "Considerations in Defining Quality of Health Care, Part I," in R.H. Palmer, A. Donabedian, and G.J. Povar, eds., Striving for Quality in Health Care: An Inquiry Into Policy and Practice (Ann Arbor, MI: Health Administration Press, 1991).
Udvarhelyi, Steven, oral testimony before the joint Subcommittees of the Advisory Commission on Consumer Protection and Quality in the Health Care Industry, June 25, 1997.
