Harriet Tyson is an education writer, researcher, and consultant. She is the author of "A Conspiracy of Good Intentions: America's Textbook Fiasco" (Council for Basic Education, 1988) and "Who Will Teach the Children: Progress and Resistance in Teacher Education," (Jossey-Bass, 1994), as well as dozens of articles and reports on teachers and teaching, textbooks, and curriculum. She is a former high school teacher and was a member and president of the Montgomery County (Maryland) Board of Education. She has worked as a writer, editor, project manager, or researcher for a number of Washington-based organizations including the Council for Basic Education, the Institute for Educational Leadership, the Association for Supervision and Curriculum Development, the Council of Chief State School Officers, and the Rand Corporation.

This paper was commissioned by the National Education Goals Panel, Purchase Order #4331FJ700011 for release at its meeting on July 30, 1997. The opinions and recommendations expressed in this paper are those of the author and do not necessarily reflect those of the Goals Panel or its members.

INTRODUCTION

The American system of textbook production and purchasing is inherently splintered. On the supply side, there are private businesses that must capture a significant share of the national market in order to remain in business. On the demand side, there are 50 states, each with a Constitutional responsibility for governing its public education system. Twenty states exercise varying degrees of control over textbooks, and thus the 15,000 school districts in the nation have varying degrees of autonomy over textbook selection. Finally, there are three million teachers, seen by publishers as their ultimate market, even though most of the books they choose have already been powerfully shaped by centralized forces in a few populous states.

This juxtaposition of a national industry and decentralized educational governance has produced a de-facto national curriculum. The irony here is that many state leaders who strongly advocate state control over curriculum willingly submit to the hodge-podge national curriculum embodied in most textbooks, including those in mathematics and science--the focus of this paper.

A Splintered Vision: An Investigation of U.S. Science and Mathematics Education,1 is the most recent research study to report that mathematics and science textbooks are as splintered as the system itself. This study has been preceded by nearly three decades of research on textbooks, virtually all showing that textbooks flit from topic to topic, covering very few of them in the depth a beginner would need to understand, remember, and integrate the knowledge.

Despite the development of national and state standards, and of standards-driven curricula and tests, the problem of superficial textbooks is still with us and appears to be getting worse, at least temporarily. But it should be said at the outset that mere topic reduction would not lead to higher levels of student achievement. The goal is not merely textbooks with fewer topics, or even lengthier treatment of "key" topics, but books with a coherent vision of the disciplines presented as an unfolding story, allowing even children in the early grades to connect the bits and pieces to larger concepts.

A Splintered Vision presents a quantitative analysis of the textbook problem and characterizes American textbooks as "a mile wide and an inch deep." It identifies textbooks as a major obstacles to higher levels of academic performance by American students. This paper presents a structural analysis of the textbook problem, raises qualitative issues, and then sets forth some recommendations.

The very structure of the textbook market, the commonly used methods of examining and selecting textbooks, and publishers' responses to those methods, affect the character and quality of textbooks in all categories. But the core disciplines are inherently different. Each one requires a different mix of fact, imagination, and theory; of process and product; of skills and concepts. Math and science, more than other disciplines, demand mastery of prerequisite skills if students are to succeed at the next highest level of difficulty. Across the disciplines, there are varying degrees of unity among practitioners about what and how to teach the young. The confluence of professional forces and market forces--to be described in the next section--therefore has varying effects on textbooks in each discipline.

THE TEXTBOOK MARKET

It is a truism that American textbooks are driven by "the market." This truism, however, belies the complexity of how the various players in this commercial/educational drama--states, school districts, teachers, book publishers, test publishers, academics, standards-setting groups, and pressure groups--interact and unwittingly conspire to maintain textbooks that are "a mile wide and in inch deep." For would-be reformers, a nuanced understanding of a multi-layered market is an essential starting point.

Adoption States

Twenty states (down from 22 a decade ago) are called "adoption states" because they "adopt" a list of state approved textbooks and bear the cost of textbooks for all students in the state. As seen in Appendix I, nearly all adoption states are in the South, Southwest, and West.

The rationale for state-wide adoption and funding has varied over the 100 years that the system has been in place. In the early 1900s, textbook purchasing at the local level was notoriously corrupt; therefore part of the impetus of state-wide adoption was to stamp out corruption. Also, widespread poverty in the agricultural states moved state leaders to provide free textbooks for students whose parents couldn't afford them. Student mobility was a problem, and the adoption states typically selected one book for each grade level and subject matter in order to ensure continuity in the education of mobile students. Finally, state leaders, then and now, believed that state selection of textbooks helped local districts who lacked the expertise and resources to make wise choices for themselves.

As state-wide adoption has evolved, so have its rationales and practices. In the 1950s, complaints from publishers about restraint of trade and appeals from local educators for more choice prompted the adoption states to adopt a list of approved books for each grade level and subject matter, rather than a single book. States began to set aside funds (usually minimal) that local districts could use for unapproved books and supplemental materials, and some established waiver procedures (usually cumbersome) by which a district could obtain state funds for off-list books. These somewhat grudging liberalizations of state policy took some of the steam out of bitter local controversies over reading pedagogy, social content, and biological evolution.

With the emphasis on accountability in the 1970s came a new rationale for state-wide adoption: if the states were to be held accountable for student performance, then it was incumbent upon state leaders to call for and purchase books that covered everything in the state curriculum and testing program. In the 1990s, with the introduction of state standards and standards-based curricula, "alignment" has become the principal criterion for textbook selection in the adoption states: a "good" textbook is one which is tightly aligned with everything in the state's instructional program.

Paradoxically, adoption-state policies have simultaneously become looser. In the 1980s and 1990s, the adoption states have adopted longer lists (more choices for local educators), or taken the cap off the number of books that can be approved, or made waivers easy to get. Florida sets aside 50% of its textbooks funds for the purchase of non-adopted books, and Texas will soon designate two approved lists, "Conforming" (100% aligned) and "Non-conforming" (50% or more aligned), and allow districts to select from either list.

These recent liberalizations in adoption state policies seem to mediate the tensions between state control and local demands for flexibility. The policy message seems to be: "We, the state, will recommend books that are aligned with state goals. If local educators believe they can meet the goals with non-aligned books, they are free to buy them, but they will be held accountable for results."

The new flexibility given to local school districts has not altered the basic pattern in the adoption states: local educators strongly prefer state-approved books because they are told that those books provide the best possible match with the state's required curriculum and testing program. The purchase of state-approved books is seen as a moral imperative by many educators because they believe it is unfair to test students on material not covered in the book. This concern with alignment has become more urgent in recent years with the rapid growth of testing programs and policies that hold teachers or schools accountable for student test scores.

Texas, California, and Florida

Some adoption states are far more attractive to publishers than others. California, Texas, and Florida offer the potential for large profits; collectively, they represent about 25% of the total national market. (See Appendix II for "Percent of Total Sales" by State) If the publisher's book can clear the adoption hurdle in one or more of those states, the company's viability is virtually guaranteed. Conversely, if a company fails to win state approval, it is shut out of the entire market in that state, and may even be forced out of business. Adoption contests are a treacherous business, especially in California. Therefore publishers study the curriculum frameworks, bid specifications, selection criteria, and politics in those states with the concentration of someone facing the prospect of an immediate hanging.

With very few exceptions, publishers cannot afford to develop a textbook tailored to any one state's demands. In deciding what to put in a book, each publishing house takes into account the aggregate demands of a handful of market areas. How each house defines this aggregation is a trade secret, but it is clear that the combined curricular demands of California, Texas, and Florida dominate the scope and sequence of nearly all textbooks published by mainstream houses.

But there are other influences as well. Although New York is not an adoption state, it is a large market and some publishers seem to give a nod to the requirements of the New York State Regents Examination. The multicultural content of textbooks in all categories reveals the influence of a few major cities in both adoption and non-adoption states.

The combined mass of facts, topics, ideas, concepts, vocabulary, cognitive tasks, pedagogical features, and social imperatives is enormous. Moreover, the mass is often riddled with internal inconsistencies because major markets want different things at different times and often project contradictory views on content, pedagogy, and sequence.

Florida, for example, wants histograms taught in 5th grade mathematics; everywhere else, histograms are taught later. In order to please Florida, but not annoy teachers elsewhere, a publisher will tuck a histogram into a side-bar activity.

Math is math, whether in Los Angeles or Tokyo, and the underlying principles and processes of science are everywhere the same. But these self-evident truths mask unresolved conflicts about what approaches to science and mathematics are best for particular students in particular places under particular circumstances.

There are legitimate resolutions of these conflicts, as seen in the publishers' earnest attempts to reconcile traditional and conceptual mathematics. But there are also illegitimate resolutions of conflicts. For example, a 5th grade life science book will feature anatomically correct drawings of the digestive and circulatory systems; but in the study of the reproductive system, the book will feature pictures of a fully clad man and woman instead of an anatomically correct drawings.

It is important to notice, however, that the big adoption states, by themselves, issue curriculum documents that contain more topics than could be respectfully treated in any standard-sized book. Thus even if the publishers could afford to produce a separate book for just one state, the book would still be overstuffed and the text would still be too compressed.

In every publishing house, the allocation of space in the book is a hard-fought battle. Authors want more space for text. Marketing departments and art directors fight for more pictures because graphics sell books. Because American teachers have less time to prepare than teachers elsewhere (another major finding of A Splintered Vision), publishers must allocate nearly half the space in the book to instructional activities. The famously thin Japanese science textbooks can be thin not only because there is a national consensus about which topics are "key" and deserve full-blown expositions, but also because Japanese teachers have time to develop their own instructional activities and materials.

There is a circularity to the national textbook market. State curriculum writers, test publishers, and textbook publishers consult one another's documents when developing new products. Those in charge of each component of the instructional program are rightly concerned about alignment, which is thought to be desirable by nearly everyone. Yet the circular motion of the overall enterprises tends to promote the accretion of topics, rather than greater focus and depth. Moreover, typical methods of textbook evaluation--discussed on the next section--rivet the publishers' attention on exhaustive, conspicuous inclusion.

In this decade, a new source of topic expansion has arrived on the scene. National subject-specialty organizations have issued standards documents, and in response, states have interpreted those national standards documents and developed standards documents and curricula of their own. Publishers have begun to respond, but cautiously. For example, probability, estimation, and graphic representation--topics inspired by the National Council of Teachers of Mathematics (NCTM), and reflected in some, but not all state and local curricula--have been added to elementary and middle-grades books. But instead of integrating these topics into the study of whole numbers (which goes on for years and years in the U.S.), they are presented separately, as though they were different branches of mathematics.

Collectively, but not intentionally, the adoption states require the creation of textbooks that break up knowledge into little pieces, the better to reflect the particulars required by the each important market. Not much space can be given over to organizing principles, or to helping students see the connections among ideas. In the case of high school biology and chemistry books, there is an overwhelming mass of unfamiliar terminology. But not much space is given to explanations that would make the terms meaningful, or to the underlying structure of the discipline. Even highly skilled readers are often defeated by these books.

There are bright spots in these textbooks, and variations in quality among them, but the prolific, ever-changing, and contradictory demands of the big state markets constrain the possibility of significant qualitative variations among textbooks. "Niche" publishers, foreign and domestic, nibble around the edges of the national market. New companies rise up like desert flowers to exploit sudden shifts in market demands, and wither as the major publishers catch up with the demand. In the main, though, blockbuster textbooks aimed at the combined big adoption state markets capture the lion's share of the vast American market.

Local School Districts

Local school districts in the 30 non-adoption states select their own textbooks, but even the most populous districts have little influence on what goes into the books. New York City, for example, is a larger market than most adoption states, but the publishers have no incentive to cater to it. Why? First, New York City does not project any particular set of demands. (If it did, the "national" textbook would be even more cluttered than it already is.) Second, virtually every book submitted to New York City survives the screening process and is "on the list." Therefore each publisher competes against all the others and can expect only a small market share.

Local Selection Mechanisms

In local school districts, committees examine the offerings and try to select the ones best suited to the district's needs. Some selectors know their subjects; other don't. Some selectors are trained for the task; others are not. Some are given adequate time to really study the books; most are not.2

In the rare cases when selectors are trained, given enough time to actually read the books, and make the very best selections, their decisions can be overturned by district administrators for reasons that have little to do with quality. District officials sometimes impose a single book in each category on the entire system, hoping for managerial simplicity and economies of scale. Often they respond to deals offered by competing companies: free teacher manuals, free training for teachers on how to use the book, free class sets, workbooks, lab manuals, and more. Free training in mathematics and science is very appealing to administrators because many teachers are poorly educated in those subjects and in-service training budgets are always vulnerable.

The cost to the publishers of mounting sales campaigns in 15,000 school districts, along with the cost of all the giveaways and inducements, makes textbooks very expensive and discourages their replacement even when better books become available.

Teachers

Teachers, either as individuals, faculty members, or members of adoption committees, often make the ultimate decisions about textbook purchases. Their influence on content coverage is small, but they have a big influence on the instructional components of textbook programs.

Publishers conduct endless pre-publication focus groups to find out what teachers like and what makes them balk. Teachers are often paid to "pilot" a new textbook and give publisher feedback, reporting on lessons that do or don't work, and saying which exercises and experiments are feasible under real-world conditions. A standards-driven mathematics book, for example, might presume that students have graphing calculators or access to spread sheet software when they actually don't; a publisher might modify the exercises in the book to take account of situations where students don't have the required technology. Through their contacts with teachers, publishers continually refine and reshape their products.

Market research by publishers on teacher preferences show that American teachers want comprehensive textbooks that cover many topics briefly. Reform groups report similar findings. For example, Project 2061's workshops and surveys of science teachers found that "relatively few teachers strongly agreed with some central reform ideas, such as 'less is more,' or that science education reform should be driven by learning goals."3

Why teachers prefer encyclopedic books is not known. But it is known that teachers are generally obedient to the demands of their superiors, and probably for that reason, they want textbooks that mirror the curricula and tests defined by those superiors.

How Teachers Use Textbooks

Decades of research on teachers' use of textbooks shows that the overwhelming majority use textbooks "as their main curriculum guide and source of lesson plans, especially teachers at the elementary school level who are responsible for five or six subject areas." 4 The data on textbook use in A Splintered Vision generally support previous studies. William Schmidt, the senior author of A Splintered Vision, observes that experienced teachers are more likely to make judgments about which topics to dwell on and which ones to skip, while teachers who are poorly prepared in mathematics and science, are in difficult teaching assignments, or are frightened by accountability mechanisms, tend to go from the front to the back of the book, skipping little or nothing. As a result, their teaching reflects the once-over-lightly approach of the books.

For the purposes of this examination of the "market," it is sufficient to note that teachers like textbooks the way they are, regardless of their degree of dependence on them.

Small Comforts, Big Sales

Textbook salesmen compete for teachers' business by providing an ever-expanding array of "extras" that make teaching more convenient and less hectic. The mainstay is the teachers manual, which provides suggested enrichment activities for advanced students, activities for slow learners and special education students, questioning strategies, tips on how to reach students with varying learning styles, and bibliographies. There are also posters, transparencies, audio and video tapes, lab manuals, instructions on how to use the book in a modular scheduling situation, and "wrap-around" kits for laboratory experiments. All of these extras cost publishers a great deal, and the cost is passed on to the buyer.

A company without the resources to produce these extras has little chance against a well-capitalized company that can offer a an array of labor-saving aids. And those companies are the very ones whose books have sold well in the big adoption states because they satisfied lengthy topics demands.

HOW TEXTBOOKS ARE EXAMINED

The criteria for selection, and the means of matching the criteria to the books, are matters of paramount importance for those who wish to improve textbooks. So a quick review of the recent history of adoption criteria, and how they are applied, is necessary to understand why the problem of unfocused textbooks tends to get worse, not better.

Evolution of Selection Criteria

Prior to the 1960s, textbook examination was a casual, in-house affair. Central office staff eye-balled the alternatives and usually selected one textbook for each grade or subject. They tended to see textbooks more as commodities than as extensions of state curricular policy or as instruments of test score improvement.

In the early 1960s, textbook selection practices changed. Black leaders in large Northern cities pressured school boards to reject textbooks that contained racial stereotypes, and to develop quantifiable methods for determining racial fairness.

Textbook selection suddenly became more public as school districts began to include parents and pressure group leaders on newly formed selection committees. Educators devised checklists with rating scales and asked selectors to judge the books' fairness to blacks, and in the years following, their fairness to women, ethnic minorities, the handicapped and the elderly. Pictures became the publishers' primary vehicle for satisfying public demands for concrete evidence of fairness.

Checklists of the 60s and 70s included other dimensions of the books as well. Typically, reviewers were asked to rate "Content" on a scale of 1 to 5, as well as "Author Credentials," Durability," "Eye Appeal, " and "Readability."

"Readability" was determined by a technical formula which was blind to both sense and style. The readability score became a make-or-break criterion for textbook adoption, and because of that, publishers began to write prose that would yield the correct score on a readability formula analysis.

These selection criteria diverted publishers' and reviewers' attention away from substance and quality, and toward matters both bogus and superficial. The checklists described above are still used in some places because they carry an air of objectivity, give the raters something concrete to do, and don't cost very much to administer. 5

Selection criteria have improved only slightly in the last two decades. For example, instead of making a global judgment about "content," reviewers are now asked to make a global judgment about "alignment." In Appendix III, a typical 1990s-style criteria sheet is compared to a standards-driven, research-based model developed by Project 2061.

The Influence of Textbook Research on Selection Practices

During the 1970s and early 1980s, there was an avalanche of research converging on the idea that many of the problems students were having with textbooks lay in the textbook, not the children. Many studies showed that the "dumbed-down" prose that had been written to survive a readability formula analysis was actually harder, not easier, for students to understand.6

Superficial treatment--the practice of merely "mentioning" most topics--was documented across the disciplines.7 Researchers also dramatically demonstrated that context--often slighted in textbooks--was crucial to both understanding and remembering. The overall organization of books and chapters, the quality of writing, the pertinence of graphics, the nature of questions, and dozens of other textbook features were shown either to support or obstruct comprehension.

This torrent of research did have some beneficial impact on textbook selection, and therefore on textbooks. California began to discourage the use of readability formulas and to focus reviewers' attention on the quality of writing. Other states and localities followed suit. Wherever school systems took textbook reviewing seriously, the training of reviewers began to include information on the educational value of well-crafted, interesting prose. When teachers began to judge textbooks on the basis of whether a student would actually enjoy reading the book and find meaning in it, publishers tried harder to make textbook prose clear and interesting. There was a tiny increase in the number of "high order" questions, and a much larger increase in the number of questions labelled as such.

But the chorus of researcher complaints about topic glut and trivial treatment didn't make a dent on publishers because their principal customers continued to churn out curriculum documents that would have taken Erasmus 20 years to teach well.

The 1970s marked the beginning of the era of accountability, and "alignment," or "congruence," or "correlation" of textbooks with state curriculum and tests became more important to state and local educators. But in this decade, alignment has become the Alpha and Omega of selection criteria, even if the means of assessing it are crude.

The logic of aligning all the components of instruction is unassailable. But the methods used to judge alignment have been, and continue to be, mechanical, superficial, and destructive. Three dysfunctional methods, discussed below, intensify the problem of too many topics and shallow coverage.

The Correlational Analysis

States and school districts require publishers to provide a correlational analysis--a document which cross-references their particular requirements with the contents of the textbook. Long strings of page citations indicate where each required topic or skill is "taught." (See Appendix IV, a page from a correlational analysis of a first grade mathematics book) Publishers routinely supply printed correlations for the adoption states, and generate them for local districts on request. When the head office cannot meet the demand, salesmen sometimes hire a private contractor to generate a quick correlational analysis for a local school district in order to clinch a sale. When districts don't trust the publisher's analysis, they often hire private contractors to conduct an independent correlation.

Few states or districts actually use these expensive and rather spurious documents to examine textbooks under consideration. (To get a flavor of how spurious they can be, a publisher establishing his claim to teach map skills cited a page with a photograph of a teacher pointing to a map.) But publishers use them against one another. The publisher with more page citations for a particular topic or skill will proudly point out that his rival's book has fewer citations, and often wins a sale through this adversarial use of correlation.

In such a climate, publishers are rewarded for conspicuous inclusion and have little incentive to imbed particulars in larger themes and structures, to waste many pages on thoughtful explanations, to incorporate compatible strands, or to invest in rich development of organizing principles.

Computerized Key Word Searches

In some places, textbook adoption committees have purchased computer programs which determine the degree of alignment between the curriculum and the books. These programs do the work of correlational analysis more independently and cheaply, but also more superficially. As in the case of correlations, the evidence of alignment rests on chapter titles, topic headings, required terminology, entries in the index and glossary, and passage length (which is taken as a proxy for depth). Adoption authorities are happy when their required material is "there," but they seem oblivious to the fact that everybody else's material is also "there."

Untrained Reviewers, Hasty Judgments

While states and school districts spend millions on textbooks, they often spend merely hundreds on the training of reviewers and the selection process itself. Textbook adoption authorities at all levels struggle to find the money to release teachers long enough to train them and given them time for thoughtful evaluation. Thus far, few of them have been able to convince their superiors that superficial reviews produce superficial textbooks; or conversely, that depth in the review process will summon depth in textbooks.

SIGNS OF PROGRESS, SIGNS OF HESITATION

The national and state standards documents have not had a direct impact on textbooks because they are usually written at too high a level of generalization to be useful to publishers. But state curriculum frameworks and bid specification documents are becoming the engines of change because they are more detailed and better lend themselves to the publisher's task: the translation of standards into day-by-day lessons and activities.

In science, emerging state curriculum frameworks show evidence of the influence of Science for All Americans, Benchmarks for Science Literacy, and National Science Education Standards, but their influence is still partial and sometimes diluted.8 Nevertheless, there does seem to be a gradual move toward coherence and publishers are responding with innovative, kit-based teaching materials for elementary and middle-school grades. High school science is proving more resistant to change.

The National Council of Teachers of Mathematics (NCTM) Standards have wide support among leaders in mathematics education and business, and most state and local mathematics leaders say that their standards and curricula have incorporated NCTM principles. Mathematics textbooks are generally thought to be more interesting to students than they were a generation ago, mainly because there is a better connection between math and its applications in the world. But many parents are unconvinced, and many teachers lack the required conceptual understanding. An idealistic interpretation of the NCTM Standards in California's last mathematics framework has left many children without computation skills, and set off still another swing of the pedagogical pendulum. It remains to be seen how far it will swing in the other direction. Because the trajectory of the NCTM approach, in California and elsewhere, is not yet clear, most publishers are still playing it safe.

Texas is in the process of developing a promising new set of curricular requirements and a better evaluation system. The emerging "Texas Essential Knowledge and Skills" science documents (TEKS) are slimmer than those they will replace because there has been an effort to make hard choices and create curricula that teachers can actually teach during a school year. The state has hired a contractor to develop criteria for judging what constitutes "coverage" of the TEKS, and to develop a training program for reviewers. Presumably, something beyond simple topic alignment will emerge.

Similarly, Florida's new 2 1/2 day training program for textbook evaluation teams will be driven by state standards and will focus reviewers attention on whether the benchmarks in the standards are actually taught, not merely mentioned.

A more substantive and analytic approach to textbook selection in Texas and Florida could have a powerful and positive influence on textbooks everywhere.

CONCLUSIONS

The higher level of coherence and challenge embodied in national mathematics and science standards reports are gradually changing the ways the states define their own standards, curriculum, and tests. While it is too early to predict the nature of a national consensus on mathematics and science education, the national standards movement seems to be a potent force in the direction of consensus. It is reasonable to expect that textbook publishers will respond to that emerging consensus over the next decade. The weakest link in chain of processes, though, is textbook evaluation. The most powerful and direct way to draw forth better textbooks is to create and sustain a well-funded, unhurried, and thoughtful system of textbook evaluation.

The National Education Goals Panel Can Encourage Better Textbooks

In a market-driven system, the only strategies that have proven successful are those that change buyers' preferences and practices. The recommendations below seek to change the market, which alone can change the textbooks.

RECOMMENDATIONS

Recommendation I: In convincing language, with concrete examples, and in every possible venue, articulate the case for more focused, standards-driven curriculum and textbooks. The public and the teachers are not yet convinced. (Examples: Many parents think that the study of estimation in the early grades is a waste of time because there are no "right answers." Many teachers believe that graphic representation is too difficult for elementary children.)

Recommendation II: Finance, or otherwise encourage, state development of selection criteria and selector training models which are standards-driven, intellectually defensible, and informed by research. A deeper and more qualitative adoption process is the single most powerful way to improve textbooks. (See Appendix V for a brief essay, "How to Recognize Quality in Textbook Evaluation." )

Recommendation III: Provide qualitative evaluations of instructional materials to local school districts. Instead of merely sending them a list of recommended books and numerical rankings representing their degrees of "alignment," distribute reports that discuss both strengths and weaknesses of analyzed materials and provide evidence to support all judgments. (See Appendix VI for an example of an excellent analytic evaluation.)

Recommendation IV: Stop demanding correlational analyses and end the use of computerized key-word searches as a method for determining curricular alignment. These methods not only drive up the cost of books, but reinforce the atomization of textbook content.

APPENDICES

APPENDIX I

APPENDIX V

HOW TO RECOGNIZE QUALITY IN TEXTBOOK EVALUATION

If you ask your local or state school administrators about their textbook evaluation process, they will probably tell you that they have a very good one. To many educators, a "good" process is one that matches the topics in the syllabus to the topics in the book, using the blunt instruments described in this paper. To others, a "good" process means a "good" checklist or rating sheet developed by a very fine and representative committee.

Even the best checklist, however, is utterly useless under the circumstances that prevail in nearly all cases. If a committee of 10 teachers is given a stack of 25 books submitted for adoption, told to "rate" each book on a scale of 1-5 according to 16 criteria, and given seven hours (a generous amount in most cases) to do the job, then about 16 minutes can be spent on each book, and one minute spent on assessing each criterion ("factual accuracy" for example). Therefore one question to put to your state or local administrators is:

How many minutes can textbook evaluators spend rating each textbook program on each criteria?

If they cannot answer your question, or sputter about the cost of release time for teachers, then there is a high probability that the process is too superficial either to select the best available textbooks or to influence publishers.

Even if a jurisdiction uses a very fine checklist and gives the evaluators several days to evaluate the textbooks, the value of the exercise is nullified if the raters lack a common understanding of the items on the checklist. Any two evaluators might have quite different understandings of an item such as "inquiry-based activity, " or "appropriate graphics." Thus another question to put to the educators in your state or locality is:

Are textbook evaluators given time to develop a common understanding of the meaning of evaluation criteria before evaluating the books?

If raters are merely indoctrinated by an administrator on how to fill out the forms, and have no time to explore examples and counter-examples of quality, then the rating scores are probably meaningless.

In the few places where textbook evaluation is taken seriously, one can observe one or more of the following:

1. Evaluators are selected primarily on the basis of subject-matter knowledge and demonstrated excellence in teaching. "Subjective" criteria, such as taste and judgment, are also important factors in choosing members of a selection committee. Where it is necessary to "balance" a selection committee according to diversity, geographic representation, seniority, union affiliation, or pedagogical philosophy, these criteria are subordinate to knowledge, effectiveness, and integrity.
2. In addition to working teachers, selection committees include true subject-matter experts (sometimes from universities and sometimes from K-12 curriculum departments), college and trade school instructors who inherit the graduates of the system, and representatives of business and industry whose employees are expected to understand the material being taught.


3. There is a division of labor on the committee: according to their talents, some members may be asked to review the books for accuracy, others for instructional soundness, others for coherence and depth of treatment, others for the quality of writing.


4. If there is not enough time to read entire books, then evaluators agree to review crucial topics in depth. Evaluators might be asked to outline a particular chapter in each book--a good check against poor organization and incoherence.


5. Evaluators are required (not merely permitted) to defend their judgments and conclusions in writing.


6. Textbooks are reviewed from the student's perspective as well as the teacher's. The evaluators address such questions as: "Would a student voluntarily read this book?" or "If a student missed class, could he reasonably be expected to learn the missed material by reading the book?" Sometimes, teachers ask students to study a chapter in a book under consideration and solicit their reactions. The "best" textbook is a waste of money and time if the students cannot or will not read the book.

1. Schmidt, William H., McKnight, Curtis C., and Raizen, Senta A., A Splintered Vision: An Investigation of U.S. Science and Mathematics Education (Dordrecht/Boston/London: Kluwer Academic Publishers, 1997)

2. Squire, James R., and Morgan, Richard T., "The Elementary and High School Textbook Market Today," in Textbooks and Schooling in the United States, Part I. D. Elliott and A. Woodward, Eds. (Chicago: University of Chicago Press, 1990).

3. Zucker, Andrew; Young, Viki; and Luczak, John, Evaluation of the American Association for the Advancement of Science's Project 2061: Executive Summary. (Menlo Park, CA: SRI International, 1969)

4. Woodward, Arthur, and Elliott, David L., "Textbooks: Consensus and Controversy," in Textbooks and Schooling in the United States: Eighty-ninth Yearbook of the National Society for the Study of Education, Part I (Chicago: University of Chicago Press, 1990)

5. Comas, Jackie, "Review of Seventy Textbook Adoption Criteria Sheets from both Adoption and Non-Adoption States," unpublished study, Indiana University, 1982

6. Armbruster, Bonnie B., Osborn, J., and Davison, A., "Readability Formulas May be Dangerous to your Textbooks," Educational Leadership, Vol. 42, no. 7, April 1985: Klare, G.R., "Judging Readability," Instructional Science, 1976, and Klare, G.R., "A Second Look at the Validity of Readability Formulas," Journal of Reading Behavior, 1976.

7. Tyson-Bernstein, Harriet, "A Conspiracy of Good Intentions." (Washington, D.C: Council for Basic Education, 1988).

8. Zucker, Andrew A., Young, Viki M., and Luczak, John M., "Evaluation of the American Association for the Advancement of Science's Project 2061: Executive Summary." (Menlo Park: SRI International, 1996).