Andrew A. Zucker, Ed.D.

Program Manager, Mathematics and Science Education

With assistance from:

• Patrick Shields
• Nancy Adelman
• Daniel Humphrey

April 1997

SRI Project 1817

This paper was commissioned by the National Education Goals Panel (Purchase Order #4331FJ700008) for release at its meeting on July 30, 1997. The opinions and recommendations expressed in this paper are those of the author and do not necessarily reflect those of the Goals Panel or its members.

INTRODUCTION

In recent years, many states have articulated clearer, more ambitious standards for what students should know and be able to do as they progress through school. However, results of the Third International Mathematics and Science Study (TIMSS), as well as results from periodic administrations of the National Assessment of Educational Progress (NAEP), demonstrate the gap between current student achievement in the United States and the higher academic standards that states are aiming for. The National Education Goals Panel (NEGP) would like to suggest methods for implementing rigorous state standards effectively so that, over time, they will help raise student achievement. The Goals Panel has therefore asked SRI International1 to describe how states have been implementing their standards. How are states using standards as they select and fund textbooks or curricula, provide support for teachers and instruction (including states' policies relating to pre- and in-service teacher professional development), and conduct assessments of students' learning? The purpose of the paper is to illustrate selected state efforts in these areas, discuss what we know about how states currently support or implement their state standards, and recommend ways to make those processes more effective.

In preparing this paper, SRI International used an extensive data set collected for prior and ongoing work. SRI is currently completing the last of 5 years evaluating the National Science Foundation's (NSF's) Statewide Systemic Initiatives (SSI) Program.2 That program has supported 25 states and Puerto Rico in ambitious efforts to improve mathematics and science education in elementary and secondary schools. Research conducted for the evaluation has focused on efforts in all the Statewide Systemic Initiatives (SSIs). However, a particularly intensive data-gathering effort has taken place in a dozen case study states. SRI has published case studies describing mathematics and science education improvement efforts in three of those states (Connecticut, Delaware, and Montana_see Zucker and Shields, 1995), and plans to publish case studies about the other nine (Arkansas, California, Kentucky, Louisiana, Maine, Michigan, New York, Vermont, and Virginia) later this year. Much of the data used to prepare this paper comes from the SSI evaluation.

Other sources of information include SRI's current evaluation of the Dwight D. Eisenhower Mathematics and Science Education State Curriculum Frameworks Projects (a program that has supported 16 states to develop and implement curriculum frameworks in mathematics, science, or both) and the evaluation conducted last year by SRI of the American Association for the Advancement of Science's Project 2061 (the first U.S. effort to develop national science education standards). A variety of secondary data sources are also referenced.

TIMSS FINDINGS: SOME KEY CLAIMS

TIMSS shows (as NAEP does) that American students are performing at levels below those that most people believe are appropriate or those that correspond to the higher academic standards established in recent years by many states. TIMSS not only compared student achievement in mathematics and science at three grade levels in 50 educational systems around the world; the study also examined the goals and many characteristics of classroom practices in those education systems. The TIMSS findings are provocative and have stimulated a good deal of discussion about efforts to raise student achievement.

Some key TIMSS claims about American education can be found in publications called A Splintered Vision (Schmidt et al., 1997), Pursuing Excellence (Peak, 1996), and Characterizing Pedagogical Flow (Schmidt, 1996). Several findings that are especially pertinent to this paper are identified below.

Curriculum

Science and math curricula (and our textbooks) include too many topics. That is, the U.S. has a curriculum that is "a mile wide and an inch deep."

Findings for some U.S. subjects/courses differ from the general findings. In particular, U.S. Algebra I classes are far more focused than other 7th- and 8th-grade classes.

There are also some important differences between mathematics and science. As an example, although the core of mathematics topics is broad, it varies little among states. On the other hand, the number of core science topics is much smaller, and the overlap among state curricula is also small. (Also, although 8th-grade American students achieve scores slightly above the international average in science, they are below average in mathematics.)

The U.S. science and mathematics curricula (including textbooks) are less challenging than in some other nations. For example, curricula in France, Spain and, to some extent, Japan reportedly pose more varied demands on students than do the U.S. curricula.

Instruction

The way that American teachers organize lessons may be flawed. American teachers act primarily as "information transmitters." Moreover, typical teachers in the U.S. use far more activities in a single lesson than seems appropriate.

Standards-based reform

Fundamental changes in various components of the U.S. education system may be needed to improve student achievement. The TIMSS reports suggest that changes are needed in teacher preparation, working conditions in schools, the quality of curricula and textbooks, and other aspects of American education.

Some comments on these claims

Virtually all the major findings are consistent with or expand on findings from other research, and this consistency adds to the credibility of TIMSS. For example, SRI convened a panel of experts to review selected science texts as part of an evaluation of the American Association for the Advancement of Science's Project 2061. Findings from that review are consistent with the TIMSS findings. In particular, the SRI review of two popular high school biology textbooks published in 1995 and 1996 found that the glossaries in each contain about 1,000 technical terms (which are also used in the body of the books) and that the authors "have not eliminated material extraneous to what is required to meaningfully convey key concepts" (Zucker, Young, and Luczak, 1996).

Thus, the TIMSS data, like the SRI research just reported, support some key principles advocated in American national mathematics and science education standards documents. For example, one important tenet of Benchmarks for Science Literacy (written by Project 2061 staff) is that it would be good practice in teaching science to spend more time on fewer important topics. This approach is sometimes referred to as the principle that "less is more" (although a better summary might be "do more of less"). That position is supported by TIMSS data showing that U.S. science and mathematics curricula try to cover a great many topics but (when compared with curricula in many other nations) sacrifice intensity of coverage, and deeper understanding, to do so.

Similarly, the National Council of Teachers of Mathematics (NCTM) standards focus attention on the importance of teaching mathematics for understanding, not just so that students commit key facts and procedures to memory. The science education standards also focus on the importance of conceptual understanding. This principle is supported by TIMSS data showing that typical U.S. lessons are not as focused as lessons in many other countries, concentrate on less challenging material, and utilize the teacher primarily as a transmitter of information.

Currently, however, TIMSS data do have some significant limitations. For example, TIMSS has not yet released information about high school students. The absence of current information from TIMSS about high schools should not lead us to ignore some of the unique problems at that level. (For example, one of these problems is that the majority of high school science teachers are specialists who seem quite satisfied with their current textbooks, despite the fact that many of the books are laden with an excess of specialized terms [for supporting data, see Weiss, 1994].) This is a topic to which we will return.

The TIMSS reports published to date typically focus on the U.S. as a whole, not individual states. This national portrait obscures variations across the states, both in current practices and in efforts under way to effect change. We know that some of the differences between state education systems are large.3 It is useful to consider which states have adopted distinctly different approaches to curriculum, instructional materials, support for teachers, or other components of their education system.

LESSONS FROM STATE EDUCATION IMPROVEMENT EFFORTS

In this section, examples of distinctive and promising approaches to mathematics and science education among the states are identified. In addition, there is a discussion of the broader lessons learned from SRI's evaluation of NSF's SSI program concerning different strategies that states use to support (or "implement") standards-based education reform.

Curricula and textbooks

Instructional materials are basic to what happens in schools. The TIMSS data challenge us to consider whether typical American textbooks are likely to be effective in the hands of typical teachers. In fact, SRI found that in many of the SSI states instructional materials are still a "weak link," especially in certain domains (e.g., high school science). Only six of the SSIs focus on the instructional materials used in mathematics and science classrooms as a major part of their change strategy.

To improve student achievement, high-quality textbooks (or other instructional materials, such as kit-based elementary science programs) need to be identified (or developed, if necessary), and decision-makers need to be well informed about them. Whether decisions are made at the state or the local level, we believe it would make sense for more states to adopt and/or disseminate information about existing high-quality curriculum materials. In a few cases, the SSIs have engaged in ambitious efforts to create new instructional materials. One successful example can be found in Montana.

Montana's SIMMS Project: High-quality materials development. Montana's Systemic Initiative for Montana Mathematics and Science (SIMMS) is producing six 1-year-long, integrated mathematics courses for grades 9-12. The materials are innovative (e.g., they assume that students have access to powerful computers and graphing calculators throughout, and they focus extensively on applications of mathematics in real-world settings), they have been highly rated by at least two sets of external reviewers, and both the utility and sustainability of the effort are reflected in the fact that the materials are being distributed nationwide by a commercial publisher. Note that schools in Montana, a local-control state, are not required to use the new SIMMS curriculum, but many do. The legislature has provided millions of dollars to fund technology for SIMMS students and teachers throughout the state.

Montana is not the only state to focus on using the curriculum to implement a new "vision" for mathematics and science education. For example, many years ago New York developed an integrated high school mathematics curriculum for college-bound students, eliminating separate courses for algebra, geometry, and other strands of mathematics, and it is still used. New York is also one of the few states to focus on technology (in the sense of engineering) as a topic that all students must study in middle schools. More recently, California developed a course called Math A as a richer option for 9th-grade students than general math, giving them an option to enroll in Algebra 1 in 10th grade. The Puerto Rico SSI has been developing a series of year-long curricula in science. Although such state-specific courses or curricula are more the exception than the rule, they do demonstrate that it is possible to innovate on a large scale and to use the curriculum as one way of implementing a state's vision of what mathematics or science education should be.4

Other SSIs, such as New Mexico's, have promoted existing high-quality instructional materials. Full Option Science System (FOSS) is an example of one set of elementary science materials promoted in New Mexico and in a number of other states. (FOSS, distributed by Encyclopaedia Britannica, was developed with NSF funds.) New Mexico is a textbook adoption state, but it is probably not necessary that a state be one for state education agency staff to recommend specific curriculum materials to schools and districts.

Over the past decade, the National Science Foundation, a few private foundations, and some commercial publishers have invested many millions of dollars in the development of various curriculum alternatives in mathematics and science.5 Increasingly, it should be possible for states, districts, and schools to choose and use instructional materials that differ from the mainstream in important ways(if they wish to do so. Ideally, professional development for teachers should be part of the effort to change curriculum and instruction, because new curricula typically require that teachers learn a great deal (both new mathematics or science content and new teaching techniques, such as using technology) before they can use the instructional materials effectively.

Teaching and professional development

The TIMSS data show not only that American curricula need to be reexamined but that typical patterns of instruction should change to be better aligned with standards of good practice. TIMSS suggests that professional development for teachers is a necessary, but not sufficient, priority if current U.S. student achievement patterns in mathematics and science education are to be substantially improved.

SRI found that professional development is the most common SSI strategy (a high priority in 18 of 25 SSIs), and in our view the majority of the SSIs using this strategy can be rated as strong or very strong in the way they conduct professional development. Delivering high-quality professional development is something that we as a nation know how to do_at least for those teachers who volunteer and when there are sufficient funds.

The SSIs used three general approaches to professional development: local human resource development, local system capacity building, and state system capacity building. The SSIs typically used a mix of these approaches, and at least nine employed all three approaches. California provides an example of a very strong professional development strategy.

California's teacher networks. California has established two teacher networks: the California Science Implementation Network for elementary science, and Mathematics Renaissance for middle school mathematics. Both networks were created before the state applied for an SSI award, and both are aimed at putting into classrooms the kind of mathematics and science curriculum and instruction envisioned in the California curriculum frameworks. The quality of the professional development is high; the programs are open to all teachers in the state; and tens of thousands of teachers have been served. Funds have been secured to maintain the programs at some level after SSI funding ends, and there is clear evidence of positive impacts on classroom practices.

Nonetheless, in spite of the strong efforts made by most of the SSIs, SRI found that the professional development system, such as it is--including state, district, and school policies related to professional development--is in need of restructuring in most states. Only a few SSI states are making efforts of this kind.

The Michigan Statewide Systemic Initiative: An example of an effective strategy for creating an infrastructure for building teachers' capacity. Michigan's SSI has focused on improving the system through which high-quality professional development is available to all local school districts. The MSSI has conducted professional development for providers of professional development, developed guidelines and taken action to restructure the state's system of professional development, and helped revise professional development standards.

Massachusetts is trying to change its professional development system. A new law in Massachusetts (not directly tied to the SSI) has radically altered its recertification policies by requiring all teachers to write individual professional development plans. The state also specified the types of professional development activities teachers need to participate in to receive credits toward recertification. (The jury is still out in terms of how much impact these changes will have on teachers and teaching.)

These two examples are the exceptions. As long as the professional development systems in the states remain unchanged, policy-makers will face difficult trade-offs between using available funds to work with large numbers of teachers superficially or with small numbers (usually volunteers) intensively. The SSI evaluation team believes that professional development should be better embedded in the work life of all teachers, as part of their job. A variety of new technologies (e.g., interactive CD-ROMs, the World Wide Web) may make this integration more feasible than in the past, but it is still necessary to find time and money within the system to support professional development for teachers and administrators.

Assessment systems

The TIMSS reports (like many earlier ones) suggest that other components of the education system need to change, besides curriculum and instruction. Testing and assessment systems are a component of particular interest to the Goals Panel and to the National Governors' Association.

One basic premise in designing a system to assess students' learning is that there should be a good match between what is being tested and the important goals established for what students should know and be able to do. (In other words, performance standards established for state tests should be aligned with the goals for students that are typically expressed in state standards or curriculum frameworks.) There are numerous examples of states making significant changes in state tests or assessments to reflect changes in their goals for student learning, some of which began many years ago.

Changing the assessment system in New York. About a decade ago, predating the SSI by many years, the state of New York began using a performance-based assessment of elementary science at the fourth grade. There is evidence that this approach increased attention to the teaching of science in elementary schools, including the use of hands-on science instruction of the type promoted in the national science education standards. New York is also proposing that all students, not just those who are college bound, take the Regents examinations. Those examinations are now being revised to better reflect the state's new curriculum standards.

In spite of examples like this, many state assessment programs still do not include science, which has not been considered a "basic" subject. Moreover, in many states there is a mismatch between the goals established for students in state standards or curriculum frameworks and the content of state-mandated tests. In 1994, when SRI studied the assessment systems in the 25 SSIs, we were able to find only 6 state assessment programs in mathematics and 4 in science that met our criteria for suitability in evaluating the progress of the SSIs. Although a number of states have improved the alignment of their assessments and standards since that time, it is clear that much work remains to be done to improve state assessment systems. The commitment of Governors and business leaders a year ago to develop appropriate assessments provides a boost to these efforts (National Governors' Association, 1996).

Changing assessment systems is difficult work. In California, an innovative state testing system (the California Learning Assessment System, or CLAS) was abandoned under political pressure. One underlying reason appears to be that the standards the state established for mathematics and science education reform were not widely enough supported to sustain the assessment system (which was also expensive). There are important lessons here about mobilizing public support for reform to which we will return.

Other strategies for supporting standards-based school improvement

As part of the SSI evaluation, SRI identified eight strategies that account for the great majority of the activities supported by the SSIs (see Exhibit 1). These can be clustered into two groups, loosely corresponding to what are known as "bottom up" and "top down" strategies for school improvement (Zucker, Shields, Adelman, and Powell, 1995; Zucker and Shields, 1997).6 Several of the strategies_including support for teacher professional development; developing, disseminating, or adopting instructional materials; and aligning state policy_have already been discussed because they correspond to topics (such as standards, curricula, and textbooks) emphasized in TIMSS publications.

Strategies focused on teachers, classrooms, and schools:

1. Supporting teacher professional development

2. Developing, disseminating, or adopting instructional materials

3. Supporting model schools

Strategies focused on districts, regions, and states:

4. Aligning state policy

5. Creating an infrastructure for capacity building

6. Funding local systemic initiatives

7. Reforming higher education and the preparation of teachers

8. Mobilizing public and professional opinion

The most common strategy the SSIs have used to support standards-based reform, as noted above, has been to provide teachers with additional professional development opportunities, often during the summer. In addition to professional development, two other common strategies used by the SSIs focus resources on activities at the school and classroom level ("bottom up" change strategies): developing, disseminating, or adopting instructional materials (see the Montana example above); and supporting model schools.

In addition, five strategies used by the SSIs aim primarily at effecting change in districts or at the state level ("top down" change strategies). These five are: aligning state policies (which includes developing or revising standards, curriculum frameworks, and state assessment systems); creating an infrastructure for building the capacity of local schools; funding local systemic initiatives; reforming higher education and the preparation of teachers; and mobilizing public and professional opinion in support of school improvement efforts.

Examples of effective use of four of the strategies have already been provided in this paper (supporting teacher professional development in California; developing, disseminating, or adopting instructional materials in Montana; creating an infrastructure for capacity building in Michigan; and aligning state policies in New York and Massachusetts). Below are examples of effective uses of the remaining four.

Supporting model schools.

Just five SSIs use a model schools strategy. Members of the SSI evaluation team believe that this is a high-risk strategy. It requires a very careful plan for dissemination and scaling up. If such a plan is not well designed and well implemented, the result is, at best, improvement in a handful of schools statewide. In the states that we believe have strong model schools strategies, that effort serves as part of a much broader reform strategy.

Puerto Rico: Successful model schools. The Puerto Rico SSI established model sites in each of the seven education regions of the island. The sites were test beds for Puerto Rico's ambitious curriculum development efforts and for site-based management ("school empowerment"). In addition, they became dissemination centers in the successful process of scaling up to more schools. These efforts are all closely tied to education reform legislation that predates the SSI.

Funding local systemic initiatives.

Nine SSIs, mostly in local-control states, chose to support local systemic initiatives. The key factors in building strong local initiatives are sufficient support for participants at the local level and some kind of quality control mechanism. An example of a successful use of this strategy can be found in Maine.

Beacon Sites: Strong local initiatives. The Beacon Sites in Maine (some of which are schools and some districts) have been a core element of the SSI. By keeping the number small (seven or eight), it was possible to provide substantial support to each. Support for full-time math and science specialists at each site has helped assure the effectiveness of the strategy. The specialists have served as classroom-level coaches and have acquired, reviewed, and established libraries of curricula and instructional materials. As planned, the Beacon Sites have also been an important component of scaling up reform, gradually establishing reputations as regional centers for learning about high-quality mathematics and science instructional materials and as places that provide models of effective classroom practices.

Reforming higher education.

Although 14 of the SSIs have adopted this strategy, none of the efforts can be rated as very strong for reform of teacher preparation programs. Changing teacher preparation has been a challenge, in part because of the difficulty of changing institutions of higher education and in part because of the fact that relatively few of the SSI resources have been used for this purpose. One fairly strong effort is found in Arkansas.

The Arkansas "crusades": Involving higher education. All of Arkansas' 14 institutions of higher education are involved in the math and science "crusades" that are the centerpiece of its SSI. Although progress has not been uniform in all 14, there have been changes in liberal arts courses (mathematics and science) in both public and private colleges and universities.

Mobilizing public and professional opinion.

Many surveys and studies provide evidence that public and professional opinion is critically important for education reform. A total of 10 SSIs place a significant emphasis on mobilizing opinion. One that is worth noting is in Louisiana.

Louisiana: Mobilizing public opinion in local communities. One member of the Louisiana SSI staff devotes all his time to helping local districts inform their communities about reforms supported by the SSI. On the basis of a strong theory of change that involves altering the generally low opinion of education held by the public in Louisiana, the SSI responds to local media requests for information and provides custom audio, video, and written information about how the SSI is improving math and science education in the state. The SSI has helped local entities to develop high-quality presentations that are reported to have had a positive impact in their communities.

Each of the eight strategies shows certain strengths and weaknesses (Zucker and Shields, 1997). For example, several of the strategies (especially aligning state policy, reforming teacher preparation, and mobilizing public and professional opinion) typically require political, technical, and financial resources beyond the reach of a 5-year initiative funded mainly with "soft" money. One result is that few SSIs have invested heavily in the reform of teacher preparation, and progress in this area has been limited.

A key point is that the SSIs use several strategies in combination (typically about four of them), basing their selection of strategies to use on the state context (e.g., whether the state has a strong tradition of local control of schools or more centralized control), the perceived needs in the state, the strengths of the staff, and other factors. Good strategic thinking seems to be an essential component of standards-based education reform, and knowledge about how to think and act strategically in the context of state education systems has grown very considerably within the past decade.

It is interesting to consider two governance issues with respect to the SSIs. First, in many of the SSIs, institutions outside of state government have taken on leadership roles. For example, in Connecticut, the Connecticut Academy for Education in Science, Mathematics and Technology was formed specifically to lead the SSI, and similar organizations were formed in Vermont and Maine. The involvement of strong institutions outside of government can be a considerable asset (e.g., by providing a leverage point that is insulated from state politics and state agency hiring restrictions or that may be able to work more readily with business groups), but what arrangement is best varies by state.

On the other hand, with limited exceptions (such as in Arkansas), few school superintendents, school board members, or other district representatives have been actively involved with the SSIs. Since many key policy decisions are made at the district level, this lack of involvement can lead to problems.

To sum up, virtually all the states receiving support from the SSI program have put into place new or improved standards for what students should know and be able to do in mathematics and science. Once that has been done, "implementing the standards" in a state means designing and effectively carrying out a coherent and strategic set of activities that support school improvement efforts based on the state's standards and its traditions. There is no formula that describes the steps that are needed, since no one knows exactly what they are. The strategies in Exhibit 1 identify the varied types of activities that are being used to support state education standards.

REFELECTIONS AND CONCLUSIONS

TIMSS is a massive study; five books are already in press to report on TIMSS findings, and more are scheduled. Similarly, the findings from SRI's evaluation of the SSI program, including the state case studies, will fill hundreds of pages. Many other excellent sources of information about standards-based education reform can be found (and some are cited here). Distilling all this information into recommendations for the Goals Panel would be a daunting task for anyone. Therefore, the title of this paper, as well as of this section, refers to "reflections" about standards-based reform. These reflections, which are based on research, are intended to stimulate debate and conversation, but certainly do not represent all the pertinent studies or all points of view.

The section begins with a discussion of the overall support that TIMSS provides for continuing efforts in the U.S. to carry out standards-based education reform. The next two sections discuss components of the education system about which TIMSS has a great deal to say, namely, curriculum (including textbooks) and instruction. Finally, the concluding section discusses a topic that undergirds all efforts at standards-based reform: mobilizing public and professional support for changes in the education system.

The need for standards-based reform

By documenting deficiencies in several components of the American education system, TIMSS provides additional reasons for believing that it is important to carry out standards-based reform. Those who are not satisfied with current levels of student achievement in the U.S. can (again) find evidence that improving the situation is likely to entail, at the least, a clarification of goals for students' learning, revisions in curricula and textbooks, and better instruction by teachers. By implication, even though TIMSS does not present evidence about state assessment systems, preservice teacher preparation programs, teacher certification and recertification requirements, state education infrastructures (such as regional centers and teacher networks), or many other features of education systems, one can conclude that these components, too, should be aligned with goals for students' learning. This is an ambitious agenda.

Research conducted by SRI, and by other groups, suggests that both "top down" and "bottom up" strategies are needed to bring about the kinds of improvements that are desired.7 Creative and strategic thinking is necessary to design and carry out a series of linked strategies that support changes in state education systems.

The starting point for standards-based reform in many states is creating a set of ambitious standards for what students should know and be able to do. Although it is clear that most states have been actively working on their standards, it should not yet be taken for granted that the standards are uniformly of high quality across the states. Research conducted by SRI and others shows that in some states the standards are clear and readable, but in others they lack internal coherence, are poorly formatted, are susceptible to misinterpretation, or are otherwise of lower quality. (To monitor the quality of state standards, the Goals Panel has supported creation of an independent entity to review them.)

Just as there are variations across nations and states in the nature and the quality of goals set for students' learning, there are important variations in curriculum, teacher professional development, assessment, and other components of the education system. One can_and should_learn from these differences across the states, just as one can learn from different nations' practices.

However, we know that a national education system can best be understood as a constellation of different features embedded in a particular culture, not just as a logical outgrowth of some particular feature (such as the nature of the textbooks in use). Thus, at the same time that we have much to learn from the TIMSS data, we must be thoughtful in reaching conclusions about what features of other education systems we wish to adopt and how those features would work in an American context. Similarly, state practices don't always travel well. Some practices are more particular to local-control states, or small states, or textbook adoption states, or states with many low-enrollment high schools (where teachers more often must be generalists), and so on. Again, this fact underscores the importance of good strategic thinking, based on the state context.

What is common across the states is widespread public support for ambitious standards and good instruction. Yet it cannot be taken for granted that educators and the public will support specific changes in the schools that state policy-makers support. We will return to this topic in the concluding section of the paper.

Reviewing textbooks and curricula

The TIMSS publications cited at the beginning of this paper heavily emphasize issues related to mathematics and science textbooks and curriculum. Although many efforts are under way to change and improve textbooks and other instructional materials, the texts used in most classrooms are as described in TIMSS publications.

Some people believe that districts, schools, or even individual teachers should be largely responsible for creating their own instructional materials. The fact is that creating a high-quality curriculum is difficult, specialized work. Some teachers have a talent for such work, but most do not_nor do they necessarily have an interest in writing curriculum, or the time to do so. Expecting most teachers to create their own curricula is like expecting most musicians to compose the music they play or most actors to write the works that they perform.

Ambitious plans for having local agencies write their own curricula are usually doomed to fail, particularly in the absence of rigorous quality control mechanisms. Such unsuccessful attempts have been a problem that SRI and NSF have documented in some of the SSIs. Other, independent studies have reached similar conclusions. A corollary of this finding is that it proves unrealistic to expect that general state standards or curriculum frameworks will provide sufficient guidance so that district or school staff can "fill in the blanks" to create their own curriculum (Massell, Kirst, and Hoppe, 1997). In simple terms: most teachers will continue to rely on textbooks most of the time.

In light of the undisputed importance of textbooks in shaping curriculum and instruction, it is surprising that there are virtually no publicly available reviews of textbooks, or other sources of information about their features or their quality_at least in mathematics and science. It is far easier to find both objective information and expert reviews about cars, restaurants, and many other products and services than it is to learn about the variations in coverage or the perceived quality of different textbooks. (In at least two SSI states, Ohio and Delaware, information about some instructional materials is being made available through the World Wide Web. However, the focus is on short units and modules more than on full textbooks or multiyear text series.) Although it makes sense to think that experts should be involved in researching and creating reviews and product descriptions, it also seems appropriate that the information should be easily available to school board members, teachers, parents, or anyone else with an interest in curriculum. This is an area in which independent, nongovernmental groups might make a contribution. Increasing the amount of information about products doesn't eliminate variation in preferences, nor does it guarantee that low-quality products will disappear. Nonetheless, increasing the number of sources and the accessibility of credible, well-designed information about textbooks could prove useful to many people.

Improving instruction

In practice, textbooks, curriculum, and instruction are closely intertwined. Changing textbooks often requires that teachers learn new instructional practices. In any event, TIMSS, NAEP, and many other studies suggest that it is important to change current patterns of instruction.

The TIMSS authors claim that there is a "characteristic pedagogical flow" that varies from one education system to another. One TIMSS publication, Pursuing Excellence, reports the results of a study that used videotaping to examine mathematics teaching in more than 200 classrooms. The findings are not encouraging.

American teachers of 8th-grade mathematics rarely developed concepts; instead, they focused their efforts on having students develop and practice individual skills. When experts reviewed the videotapes and rated their quality, 87 percent of Japanese lessons and 60 percent of German lessons were judged to be of medium or high quality, compared with only 13 percent of American lessons. Exhibit 2 reproduces a figure from Pursuing Excellence that illustrates the different approaches in these nations.

The emphasis on understanding is evident in the steps typical of Japanese eighth-grade mathematics lessons:

Teacher poses a complex, thought-provoking problem.

Students struggle with the problem.

Various students present ideas or solutions to the class.

Class discusses the various solution methods.

The teacher summarizes the class's conclusions.

Students practice similar problems.

In contrast, the emphasis on skill acquisition is evident in the steps common to most U.S. and German math lessons:

Teacher instructs students in a concept or skill.

Teacher solves example problems with class.

Students practice on their own while the teacher assists individual students.

The TIMSS findings about instruction are important, and they provide some useful guidance on how instruction in U.S. classrooms might be improved. TIMSS reminds us what a central role the teacher plays in orchestrating instruction. Videotapes of typical lessons in different countries might be quite instructive to teachers_and, in fact, the U.S. Department of Education has produced a CD-ROM showing 8th-grade mathematics lessons from the U.S., Japan, and Germany (with subtitles in translation). However, we must also keep in mind that instruction balances a great many variables. Efforts to change instruction to align better with national and state standards will involve not just focusing on key topics (a major focus of research in TIMSS) or increasing the emphasis on teaching for understanding of concepts (instead of simply memorizing facts or drilling on skills), but attending to a wide range of other variables, such as using materials (including computer technology) effectively, organizing some instruction so that small groups of students work together, increasing the number of lessons with an interdisciplinary focus (e.g., combining mathematics and science), spending more time on challenging material, making students more active learners (e.g., engaging them in the classroom discourse), etc. Teachers need a considerable amount of time to learn to teach in ways that are significantly different from their current practice, particularly when instruction involves topics that the teachers are still learning to understand themselves, as is often the case.9 It would be an error to underestimate the magnitude of the task.

The good news is that there are many successful efforts to provide teachers with high-quality professional development. Policy-makers can increase the opportunities available to teachers and better integrate professional development into teachers' normal work lives. Also, much is already known about how to make professional development effective (Corcoran, 1995). But some of the issues highlighted by TIMSS are seldom addressed in the U.S. For example, according to the TIMSS authors,

U.S. teachers lack the long and carefully mentored introduction to teaching that Japanese and German teachers receive. _ [Also,] U.S. and German teachers do not have the rich informal opportunities to learn from each other and to share questions about teaching-related issues that are enjoyed by their Japanese colleagues. (Peak, 1996)

Somehow, time needs to be found for teachers to work together to improve instruction. When time and money are available, well-designed professional development can have an impact.

A number of the SSIs have been able to demonstrate that classroom practices have changed as a result of their efforts. The changes may be more modest than had been hoped for and typically involve a specific set of classrooms heavily involved with the SSI, not whole states, but the improvements are real. Linking professional development explicitly to instructional materials that teachers will later use in the classroom is one practice that many of the SSIs have used successfully. Efforts to change instructional practices need to be systemic in other ways, too. Focusing on one teacher or classroom at a time is unlikely to succeed (because textbooks, tests, state standards, and other factors need to be aligned with and support good instruction). What is more, there needs to be strong public support for the changes in curriculum and instruction about which teachers are learning. This is an issue of continuing importance.

Mobilizing public and professional support for change

Standards-based reform has motivated thousands of talented individuals and many dozens of different agencies and institutions across the U.S. to improve education for elementary and secondary students. The development of standards, learning goals, and curriculum frameworks is receiving much greater attention now than a decade ago. Setting higher standards has also produced tangible gains for students. For example, the percentage of graduating high school students completing the minimum academic program recommended by the National Commission on Excellence in Education increased from 2.7 percent in 1982 to 29.2 percent a decade later--a 10-fold increase10 (National Center for Education Statistics, 1995).

At the same time, it is evident from a great deal of research that standards-based reform is difficult, slow work. The difficulty of standards-based reform stems from the fact that it presents not only technical but also political challenges.

In the political arena, it is critical to obtain substantial public and professional support for change. The "vision" that TIMSS speaks about must be carefully and patiently articulated by policy-makers. Some SSI states have found that it takes years to create a vision for change and obtain "buy in" from teachers and the public. Yet, without patience, there is a real risk of backlash, as has happened in a few states.

One clear lesson is that policy-makers should not try to swing the pendulum to extremes, because if people have a perception that new standards or classroom practices are too extreme, they won't support the new approaches. Research shows that good teachers will integrate old and new practices_and appropriately so (see, for example, Knapp et al., 1995; Massell et al., 1997). The public needs to understand that setting new goals (such as having much higher proportions of students successfully solve challenging problems in mathematics and science) does not mean abandoning all the old goals (such as having students memorize certain key facts). Some of the improved approaches to instruction that are called for in national and state standards, such as use of educational technology by teachers and students, will be altogether new to many parents. As older education goals, standards, and practices are replaced by new ones, it is important that the public not believe the baby is being thrown out with the bathwater.

Work done in some of the SSI states illustrates one way a synthesis can be created. In Montana--the same state that is trying a bold, distinctively different high school mathematics curriculum--policy-makers are aware that public support is vital. Montana's draft Framework for Improving Mathematics and Science Education states that:

Montanans include inquiry-based learning in their accreditation standards because they believe students best acquire scientific and mathematics knowledge when they learn in ways that reflect how mathematics and science actually work. However, Montanans are very clear about their belief that basic facts, basic skills, and basic technologies are essential tools in inquiry-based learning.

As noted earlier, many of the SSIs have set about mobilizing public and professional support for change. The SSIs have used many communication channels, including newspapers, television, and other media; public relations professionals; public forums; print publications; and even placemats in fast-food restaurants.

Public support seems to be more difficult to obtain in some areas than in others. At certain levels in certain disciplines (e.g., elementary mathematics), there are many schools and classrooms where a new vision for educational practice is being implemented, to one degree or another. However, in other areas there are fewer examples of a new vision. Among the SSIs, for example, very little effort is expended in aggregate on changing curriculum and instruction in high schools, especially in science. It would be wrong to conclude that there isn't a need for change in high schools. It simply appears to be more difficult to effect change at that level. Both educators and important segments of the public appear to resist many proposed ideas for changing high schools. For example, SRI's research suggests that changing the curriculum for students in high schools' lower tracks is more acceptable to many parents11 than changing the curriculum for college-bound and honors students (even though state and national standards call for curriculum changes that would affect all students).

CONCLUSIONS

The release of the first several of many TIMSS publications is a stimulus to consider the strengths and weaknesses of standards-based education improvement efforts. SRI's evaluation of the SSI program provides a rich database of pertinent information.

There is evidence that many of the SSIs have been able to have a positive impact on student achievement. The numbers of students involved and the magnitude of the gains are both probably smaller than one would wish. Still, these data provide more evidence that sustained attention to standards-based reform pays off. Furthermore, many of the SSIs have changed state education systems in ways that are likely to have lasting impact, even after NSF's funding for them has ended. The legacies will include revised state curriculum frameworks, improved instructional materials, continuing teacher networks, new nonprofit institutions advocating school improvement, business partnerships, attentive legislatures, and some creative, strategic thinking.

However, we have not been able to discover any simple recipe for success. The evidence shows that standards-based reform requires a variety of strategies, involving both "top down" and "bottom up" activities conducted by many different agencies and institutions. Teacher professional development is clearly a necessary part of the mix. Public and professional support for change is essential, and strategies for mobilizing such support are therefore important. The fact that changes in many different components of the education system must be addressed was stated clearly by the Governors and business leaders at last year's National Education Summit:

We recognize that better use of technology, improved curriculum, better-trained educators, and other changes in the organization and management of schools are necessary to facilitate improved student performance. (National Governors' Association, 1996)

Both state governments and the federal government have demonstrated remarkable staying power on the issue of raising education standards. A number of indicators demonstrate that progress is being made. However, TIMSS reminds us that what we are really talking about is a major cultural change involving the behavior of tens of millions of Americans. It is not surprising that the work is slow. Continued persistence is the key to success.

REFERENCES

Blank, R. K., and Gruebel, D. (1995). State indicators of science and mathematics education, 1995. Washington, DC: Council of Chief State School Officers.

Bodilly, S. (1996). Lessons from New American Schools Development Corporation's demonstration phase. Santa Monica, CA: RAND Corporation.

Consortium for Policy Research in Education. (1995, July). CPRE Policy Briefs. Tracking student achievement in science and math: The promise of state assessment systems. New Brunswick, NJ: Rutgers (CPRE).

Corcoran, T. C. (1995). Transforming professional development for teachers: A guide for state policymakers. Washington, DC: National Governors' Association.

Knapp, M. S., and Associates. (1995). Teaching for meaning in high-poverty classrooms. New York: Teachers College Press.

Massell, D., Kirst, M., and Hoppe, M. (1997). CPRE Policy Briefs. Persistence and change: Standards-based systemic reform in nine states. Philadelphia: University of Pennsylvania (Consortium for Policy Research in Education).

National Center for Education Statistics. (1995). Digest of education statistics, 1995. Washington, DC: U.S. Department of Education.

National Governors' Association. (1996). National Education Summit policy statement. Washington, DC: Author.

National Science Foundation (Division of Elementary, Secondary, and Informal Education). (1997). Review of instructional materials for middle school science. (NSF publication #97-54.) Washington, DC: Author.

Peak, L. (1996). Pursuing excellence: A study of U.S. eighth-grade mathematics and science teaching, learning, curriculum, and achievement in international context. Washington, DC: U.S. Department of Education.

Rowe, M. B., Montgomery, J. E., Midling, M. J., and Keating, T. M. (1997). ChemCom's evolution: Development, spread and adaptation. In: Raizen, S. A., and Britton, E. D. (eds.). Bold ventures, volume 2: Case studies of U.S. innovations in science education. Boston: Kluwer Academic Publishers.

Schmidt, W. H., et al. (1996). Characterizing pedagogical flow: An investigation of mathematics and science teaching in six countries. Boston: Kluwer Academic Publishers.

Schmidt, W. H., McKnight, C. C., and Raizen, S. A. (1997). A splintered vision: An investigation of U.S. science and mathematics education. Boston: Kluwer Academic Publishers.

Weiss, I. R., Matti, M. C., and Smith, P. S. (1994). Report of the 1993 National Survey of Science and Mathematics Education. Chapel Hill, NC: Horizon Research, Inc.

Zucker, A. A., and Shields, P. M. (eds.). (1995). Evaluation of the National Science Foundation's Statewide Systemic Initiatives (SSI) Program, Second-year case studies: Connecticut, Delaware, and Montana. Menlo Park, CA: SRI International.

Zucker, A. A., and Shields, P. M. (1997, February). SSI strategies for reform: Preliminary findings from the evaluation of NSF's SSI Program. Menlo Park, CA: SRI International.

Zucker, A. A., Shields, P. M., Adelman, N., and Powell, J. (1995). Evaluation of the National Science Foundation's Statewide Systemic Initiatives (SSI) Program: Second-year report, Cross-cutting themes. (NSF publication #96-48.) Washington, DC: National Science Foundation.

Zucker, A. A., Young, V. M., and Luczak, J. M. (1996). Evaluation of the American Association for the Advancement of Science's Project 2061. Menlo Park, CA: SRI International.

NOTES

1. SRI International, the former Stanford Research Institute, is one of the world's largest not-for-profit research and consulting companies. It is headquartered in Menlo Park, California.




2. SRI's partners in this effort include the Consortium for Policy Research in Education (CPRE), Policy Studies Associates, Woodside Research Consortium, and the Council of Chief State School Officers.




3. For example, in the early 1990s, California's high school science classes (average size 27) were 60 percent larger than Wyoming's (average size 17). States also vary widely in the percentage of teachers certified in the subjects they teach, as well as in many other areas of education (Blank and Gruebel, 1995).




4. The development of a full-year or multiyear curriculum demands a high level of quality control, and is therefore a task that not all states will want to take on. In Montana, for example, the development of the SIMMS curriculum has involved dozens of writers, several sets of reviewers, year-long pilot tests with students, revisions, professional editing, and other steps.




5. In the area of elementary and secondary education, one of the major goals of NSF "is to provide the field with high-quality instructional materials that incorporate the best research on teaching and learning, include accurate science and the active participation of scientists in the development process, and have undergone an extensive pilot and field test process" (NSF, 1997). As a result, in recent years, NSF has spent tens of millions of dollars supporting the development of instructional materials in mathematics and science. For example, NSF recently published results of an expert-panel review of 19 full-year and multiyear curricula for middle school science that have been supported with NSF funds (ibid.). Most instructional materials supported by NSF are published and distributed by commercial publishers.




6. The eight strategies were derived empirically, based on data collected for the SSI evaluation. This set may or may not be a complete list of possibilities. Even if it is not complete, it seems to include the great majority of actions that states have supported. (Note that efforts to increase equity in mathematics and science education cut across the eight strategies; we do not identify those efforts as a separate strategy.)




7. For example, New American Schools_formerly known as the New American Schools Development Corporation_began with the assumption that an improvement strategy focusing exclusively on individual schools would be powerful enough to change whole systems. More recently, however, a set of published lessons learned about their efforts concluded that "whole-school designs and design-based assistance are limited vehicles to reform and must be coupled creatively with other efforts," including district and state policies, public support, testing, and other elements (Bodilly, 1996).




8. The authorizing legislation for the U.S. Department of Education (P.L. 96-88) includes protections for state and local control of education, including prohibition of federal "direction, supervision, or control over _ the selection of _ textbooks." Reviews are not control, but it seems to the author that nongovernmental agencies are on firmer ground publishing information that could tread on publishers' toes.




9. Elementary teachers, for example, often have poor backgrounds in mathematics and, especially, science. Also, a significant number of middle and high school teachers are teaching out-of-field. Inadequate teacher preparation is a particularly serious problem in many poor urban and rural areas.




10. The Commission recommended that all students take 4 years of English, 3 years of social science or history, 3 years of mathematics, 3 years of science, and « year of computer science.11 A recently published case study of the ChemCom chemistry curriculum (developed by the American Chemical Society) also highlights issues related to its acceptance and use with high school students in different tracks (Rowe et al., 1997).