Pay for Performance: the worst idea ever?

Our state and federal political leaders are invested in the idea of reforming education through accountability for results. There's nothing wrong there in theory, but the implementation is being distilled down to paying (and firing) teachers on the basis of test scores. Remember the axiom: you get what you measure. (Nor is every important result easy to measure.)

So, we’re in trouble in Michigan immediately. For three years, we have invested prodigious time and effort and millions of dollars in teacher training, curriculum writing, and hardware for the on-line Smarter Balanced Assessments set to replace the MEAP in the coming school year. All of this is based on the Common Core State Standards — for which implementation the legislature just denied funding.

So, will students be tested and teachers evaluated (as mandated by state law) on the basis of old tests that do not match the new curricula? What possible sense does that make?

And what is the point? Weren’t we trying to improve student learning?

We know from a robust and valid research base what works to improve teaching and learning: when teachers focus on analyzing, together, evidence of student learning, and when they hold one another collectively responsible for these outcomes, children achieve more. See ACSD’s “How Do Principals Really Improve Schools?” for numerous citations.

A collaborative culture of teaching professionals is the research-proven best practice that we should be encouraging. Instead, our ill-informed legislature keeps mandating practices proven not to work but to actually undermine that collaboration.

“Merit pay” or “pay for performance” does not work and puts teachers into competition with one another. There is NO research showing that it improves student performance or positively changes teacher behavior.

“Merit pay” makes things worse

But beyond absolutely not working, pay for performance makes thing worse. The “bigger stick” approach to improving instruction — by putting teachers’ pay and their very jobs in jeopardy — fosters fear and short-term thinking. Fear, competition, and short-term thinking prevent rather than encourage organizational improvement.

• Daniel Pink: in the MIT incentives study, “the high reward produced the worst performance. It has been proven over and over again by psychologists, sociologists, and economists [that] incentives work for simple, if-then tasks; but when tasks require some conceptual, creative thinking, straightforward rewards like money do not work and often lead to poorer performance.”

• W. Edwards Deming: merit pay “nourishes short-term performance, annihilates long-term planning, builds fear, demolishes teamwork, nourishes rivalry and politics.”

• Large corporations proved that the “bigger stick” is a bad idea by implementing it. Both IBM and Ford, some years ago, discovered that “forced ranking” employee evaluation systems destroyed the cooperation and collegiality necessary to efficient and effective operations.

When we set out to radically change the institution of public education, we should do so on the basis of valid research proving what actually works rather than our preconceived notions. We should carefully pilot and monitor changes so as not to throw out the baby with the bathwater. And we must always put the needs of our children over our ideology or the interests of those positioned to profit off them — such as testing and software corporations and for-profit school operators.

How about applying this to the legislature?

Regarding school districts in deficit, one of our legislators was recently quoted as insisting that “Someone must be held accountable!” Given that the legislature made unprecedented cuts in school funding and diverts $400 million a year from the School Aid Fund, I’d suggest that he look in the mirror.

Or, we could apply the same outcomes-based metrics to them. An editorial (“Teacher pay proposal flunks the test”) in The Livingston Daily recently suggested exactly that:

“Michigan lawmakers are among the highest paid in the nation, but the state doesn’t have the outcomes to justify those salaries…. Here’s a plan. Cut the pay of all lawmakers in half. Then, create a salary reward system based on quantitative improvements. A 10 percent pay hike, perhaps, if the Michigan unemployment rate dips below the national average. A similar sliding-scale reward when incomes rise. How about a bonus for lowering the number of residents living below the poverty level? Or improving the health status of Michigan residents?”

Or does such a suggestion insult and outrage them? Hmmm.

How much do YOU know about schools and education?

Diane Ravitch, Research Professor of Education at New York University, Assistant Secretary of Education during the first Bush administration, and author of The Death and Life of the Great American School System, recently wrote an amazingly succinct summary of the state of the national debate about education, entitled "Do Politicians Know Anything About Schools and Education? Anything?" The ironic tone was fully intended. Because it is so inclusive of the actual, research-supported facts relevant to the debate, it deserves to be spread far and wide. She asks pertinent, if leading, questions to make us all think and rethink our positions. Because she summarizes, and her brief answers may not be believed, I will expand upon them.

Charter Schools. Ravitch asks, “Are you aware that studies consistently show that charter schools don’t get better results than regular public schools? ... That some charter schools get high test scores, many more get low scores, but most are no different from regular public schools?”

She is absolutely correct about the research. Pro- and anti-charter school folks will each cite their favored results, often from research sponsored or funded by one side, but the studies with the best research designs, which are most supportive of broader conclusions, show exactly the proportion of “results achievement” Ravitch summarizes. It is vitally important, in drawing conclusions about relative performance, to control for differences in student populations.

Charter schools do not serve the same population as traditional public schools; a larger proportion of high-needs students is left behind. Demographic data released a few days ago on the Fall 2011 MEAP tests lets me demonstrate that. Of students in grades 3–8 in Van Buren Public Schools, 54% are white, 54% are economically disadvantaged, and 9.4% have disabilities. Of students in grades 3–8 at Keystone Academy (within the borders of VBPS), 75% are white, 27.5% are economically disadvantaged, and 1.6% have disabilities. This academy serves only half as many poor children and very few children with disabilities. The charter movement is re-segregating our public schools. Remember this the next time a politician with an agenda tells you that charter schools do better with less money.

Merit Pay for Teachers Based on Test Scores. “Are you aware that merit pay has been tried in the schools again and again since the 1920s and it has never worked? Are you aware of the exhaustive study of merit pay in the Nashville schools, conducted by the National Center for Performance Incentives at Vanderbilt, which found that a bonus of $15,000 per teacher for higher test scores made no difference?”

Again, Ravitch is precisely correct. And the Vanderbilt study was carefully designed to test exactly the question of whether, with important variables controlled for, student achievement would rise with significant pay-for-performance teacher incentives. It did not. Human motivation is just not that simple. Daniel Pink elaborates on why this is so in his book, Drive. As he noted in a Washington Post interview last week, merit pay for teachers just doesn't work.

“Are you aware that there is a large body of research by testing experts warning that it is wrong to judge teacher quality by student test scores? Are you aware that these measures are considered inaccurate and unstable, that a teacher may be labeled effective one year, then ineffective the next one? Are you aware that these measures may be strongly influenced by the composition of a teacher's classroom, over which she or he has no control? Do you think there is a long line of excellent teachers waiting to replace those who are (in many cases, wrongly) fired?”

She's got the evidence on every count above. Follow her links and see for yourself. The demoralization of current teachers is obvious; the effect on the prospective teacher pool is even more worrying. Top-quality potential teachers have many more attractive fields to go into, in terms of prestige, remuneration, working conditions, security, and control over the factors upon which their performance will be judged. The sharp increase in debt for new graduates exacerbates the discouraging effect of poor job security.

Vouchers. “Are you aware that Milwaukee has had vouchers for low-income students since 1990, and now state scores in Wisconsin show that low-income students in voucher schools get no better test scores than low-income students in the Milwaukee public schools? Are you aware that the federal test (the National Assessment of Educational Progress) shows that — after 21 years of vouchers in Milwaukee — black students in the Milwaukee public schools score on par with black students in Mississippi, Alabama, Georgia, and Louisiana?”

This is the first time the voucher schools have been required to use the same state test as public schools. The voucher students performed worse in both reading and math; controlling for economic disadvantage, low-income voucher students performed about the same as those in public schools. Gov. Scott Walker's response is to propose that voucher kids not have to take that test anymore.

Internet Schools. “Does it concern you that cyber charters and virtual academies make millions for their sponsors yet get terrible results for their students?”

The internet schools do, indeed, have miserable track records. Perhaps the best evidence (in terms of numbers and number of years to show trends) comes from Colorado. An analysis shows that half the thousands of on-line students there leave the virtual schools within a year, often further behind academically than when they started. Colorado’s annual report found that “achievement of online students consistently lags behind those of non-online students, even after controlling for grade levels and various student characteristics,” including poverty, English language ability, and special education status. The limited evidence in Michigan is also not encouraging. Students at the Westwood Cyber High School in Wayne County achieved an ACT composite score of 15.6 (of a possible 36) in 2010 and 16.1 in 2011.

The American Legislative Exchange Council (ALEC), funded by major corporations and conservative foundations, drafts model bills for state legislatures. The “Private Chair” of ALEC’s Education Task Force is Mickey Revenaugh; he is co-founder and Senior Vice President of Connections Academy — “a leading national provider of virtual public school curriculum, technology and school management services,” which stands to make millions from virtual schools. I wonder where Senator Colbeck got the wording for his bill to allow a huge expansion in virtual schools?

Poverty and How We Rank Internationally. “Did you know that American schools where less than 10% of the students were poor scored above those of Finland, Japan and Korea in the last international assessment? Did you know that American schools where 25% of the students were poor scored the same as the international leaders Finland, Japan and Korea? Did you know that the U.S. is #1 among advanced nations in child poverty? ... Did you know that family income is the single most reliable predictor of student test scores? ... Affluence helps — children in affluent homes have educated parents, more books in the home, more vocabulary spoken around them, better medical care, more access to travel and libraries, more economic security — as compared to students who live in poverty, who are more likely to have poor medical care, poor nutrition, uneducated parents, more instability in their lives. Do you think these things matter?”

The disparity in incomes in the U.S. has been growing exponentially. A full third of all income growth over the past 20 years has gone to the top tenth of one percent. Meanwhile, middle-class wages have stagnated or fallen. Sharply rising child poverty is a frightening correlate to that inequality. All you need do is note the startling change in VBPS demographics in recent years to see the trend. And of course that matters when it comes to academic achievement.

The results are documented in our national results on the international PISA achievement tests. National Association for Secondary School Principals researchers disaggregated the 2010 results by income and issued a report entitled “PISA: It’s Poverty Not Stupid.” When comparing apples to apples — other nations and American schools with equally low poverty rates — our students were first in the world.

Solutions that Aren't. “Do you know of any high-performing nation in the world that got that way by privatizing public schools, closing those with low test scores, and firing teachers? The answer: none.” Nations with the best achievement records have well-trained teachers who enjoy high pay, respect, and prestige. The teachers in the best-performing nations mostly have strong unions, as well. Yet self-styled “reformer” governors all over the country seem determined to destroy unions, especially the teachers unions that did not support their election. One might be justified in questioning their motives, as well as their willingness and ability to base important decisions on real evidence.

What motivates workers?

In the spring of 2009, I remember being infuriated by a Wall Street Journal article in which it was called “poetic justice” that the United Auto Workers would “finally have a direct stake in the survival and prosperity of General Motors and Chrysler.” The reporter was referring to the negotiated payments to the VEBA (voluntary employee beneficiary association) for UAW retiree health care being paid in stock rather than the agreed-upon cash, due to impending bankruptcy. I (not a UAW member nor a relative of one) found that opinion both stupid and insulting. In what way did auto workers not have a direct stake before? Their pay/pensions and benefits have always been dependent on the prosperity of the companies, and most of them held (now worthless) stock, as well — because they believe in and are proud of their work! When you are intimately involved in making tangible products that you, your friends, and your neighbors use every day, you are both literally and figuratively invested in their quality and in the success of the company. I think that personal stake is something not well understood by folks far removed from manufacturing. Pride is a powerful motivator.

What has this to do with education? It is now widely assumed that educators can be spurred to better performance by that universal motivational tool: money. The federal Race to the Top program of competitive grants specifically insists that teacher pay and job security be linked to performance — and that performance is to be measured largely by student performance on standardized tests. All over the country, teachers are being urged to give up tenure in favor of the possibility of much higher pay, should their charges’ test scores improve dramatically. The underlying rationale for this “pay for performance” is that teachers will thus be motivated to work harder at their jobs. This assumption, like the WSJ assumption that auto workers did not care about their companies or products, is also stupid and insulting. I cannot imagine that anyone goes into public school teaching for the money. The pay starts out low and, although it then doubles as experience and effectiveness increase, it also tops out at a relatively low ceiling given the level of responsibility and the number of degrees earned by most teachers.

No, teachers, I feel quite safe in generalizing, had less mercenary and more altruistic motives for going into their field. While it is possible that low pay can discourage and disaffect them, it does not logically follow that higher pay will energize and encourage them. Human motivation is simply not that one-dimensional. Once we get past a certain level of daily subsistence, we are much more satisfied by the intrinsic rewards of work — doing something well, making a real difference, growing in expertise, feeling useful and productive — than by how much we are paid for it. That is why so many materialistic Americans reach mid-life thinking “Is that all there is?” despite having reached their goals for acquiring all kinds of expensive stuff. The notion that more money will result in better teaching is insulting because it assumes teachers are both lazy and materialistic.

And now it has been proven wrong, as well.

The National Center for Performance Incentives at Vanderbilt University (funded by a $10M grant from the U.S. Dept. of Education) employs “specialists in social and behavioral science, statistical analysis, economic theory, and policy analysis” to conduct “randomized field trials and evaluations of existing pay-for-performance programs” in public education. The report on one such study, the Project on Incentives in Teaching, or POINT, was released last week. This five-year study and analysis of a three-year randomized trial examined the effects on student outcomes of paying eligible Nashville teachers bonuses of up to $15,000 per year for increasing their students’ scores on the Tennessee Comprehensive Assessment Program tests.

The bottom line? “We sought a clean test of the basic proposition: If teachers know they will be rewarded for an increase in their students’ test scores, will test scores go up? We found that the answer to that question is no.”

This does not mean that teachers should not be paid more, but it should cause us to reconsider the simplistic pay-for-test-scores plans. If we really want to apply business principles to the improvement of public education, we should go back to W. Edwards Deming, who considered pay for performance one of the “Seven Deadly Diseases.” Instead, Deming suggests a continuous cycle of collaborative planning, implementation, evaluation of results, and adaptation of the plan in order to truly change an organization. And, imagine what being treated like professionals in this way would do for teacher morale and motivation!

May 2011 addition: you have got to see this animation summarizing Dan Pink’s Drive work on motivation and incentives!

Merit Pay for Teachers

Should K–12 teachers be paid according to how well their students learn? This is one of those ideas that sounds like a no-brainer. After all, aren’t other professionals routinely evaluated and compensated in some way that relates to their accomplishments? Why shouldn’t teachers, too, be rated on their effectiveness and paid more when they do a better job?

The problem, it seems to me, is how their effectiveness can be fairly evaluated. Looking at student outcomes seems reasonable, but that is not simple. We tend to oversimplify it by judging what children learned on the basis of achievement test scores. Most would agree that absolute score differences (just higher scores) are no measure of teacher effectiveness. Obviously, not all students of the same grade begin the year at the same level or possess the same skills. If we used that measure, everyone would fight to teach only the most gifted and motivated students.

So, should we use a growth or value-added model, to see whether individual children show a year’s progress in a year? I ran across a very entertaining video in which cognitive psychologist and University of Virginia Prof. Daniel Willingham describes — in less than three minutes! — six problems (some conceptual, some statistical) with evaluating teachers by comparing student achievement in the fall and in the spring. Among them are biased data when some children move away during the year, the effect of others on how teachers do (such as the help or hindrance of a building administrator), and the effect of peers — is the classroom cohort rowdy or well-behaved? I’ve spent enough time in classrooms to know that sometimes there is a student who seems to suck all the oxygen out of the room, making life difficult for everyone else. I might add that some years (when going through a messy divorce, or when grieving the loss of a child or a spouse) will find a typically great teacher unable to think or function as well as usual. Should she be fired? Are her students permanently harmed? I think of the lessons in compassion and in dealing with difficult people that I and my children learned from such sub-optimal school years — which reminds me of all we learn that is useful in life but not evaluated on tests.

A related conceptual problem with using test scores is that they focus on short-term gains. Most of us, in higher education and then in adult life, did not find that only discrete bits of information were important. Broader knowledge and skills — and especially the abilities to research, evaluate, learn, and apply things on our own — were much more vital to our success. Yet these skills are not evaluated on most tests.

Portfolios of student work give a superior picture of student abilities, but they would be very expensive and time-consuming to evaluate in a fair way. If we cannot find the time or money for essays in our achievement tests, then, realistically, we’ll never use portfolios.

This hints at the major problem with test results as a measure of student learning: in order to make massive testing programs at all affordable, we have made them much less valid over the years. Our MEAP writing test, for example, used to require legions of low-paid evaluators. Not only was this very expensive, and so time-consuming that Michigan was penalized by the federal government for being late with the scores (under No Child Left Behind rules), but it proved nearly impossible to get consistent grading from so many evaluators. So, the writing test today involves very little actual writing.

You get what you measure, especially if the tests have high stakes, and we are not measuring the complex skills that our children need for life success. Even the gains we do measure can be evanescent, disappearing soon after the tests were scored.

Professor of education at California State University, Monterey Bay, Nicholas Meier (rightly) fears that basing teacher pay on test scores will encourage even more focus on test fodder and test preparation alone, at the expense of more worthwhile endeavors. I recently heard an impassioned speech about education outreach plans for the Yankee Air Museum that involved teaching children forgotten history and engaging them in exciting activities, since schools no longer do. My reaction was a sad acknowledgment that, yes, if it’s not on the MEAP, our teachers don’t have time for it anymore. We trade hundreds of students (and their indispensable funding!) back and forth every year with charters and schools of choice — and those decisions are most likely made on the basis of test scores. The draconian penalties levied by NCLB are base largely on test scores. And now we want to raise the stakes by basing teacher pay and job security on them too?

Dr. Meier says this better than I could: “There is an axiom in the social sciences known as Campbell’s Law that says that the higher the stakes on a particular social indicator (e.g., a single test score), the more the use of that indicator corrupts the original intent, as it encourages people to manipulate the system to look good on that indicator regardless of other effects. We see that happening already—retaining students so they take the easier test; pushing kids to disappear from the system. There is the focus on the kids that show the most promise of moving from one category to the next, while ignoring others. Not to mention the examples of out and out cheating….”

So, no, I have little hope that pay for performance will improve public education.

There are other ways to evaluate teachers, of course, with classroom observation being a time-honored one. This tends to be more of a good idea than a good practice, though, as it is done too rarely for validity. As a young teacher in my family put it: “Who will be doing my evaluation? Will it be an administrator who has classroom experience? If so, will that be experience in my field or something totally unrelated? Are those responsible for my evaluation familiar with the standards the students are expected to achieve? If not I would question, seriously, the reliability of their judgment.”

When you think about it, there is a good reason many teachers are suspicious and fearful about the performance-based evaluation required by the federal Race to the Top education funding program. If they are unconvinced of the fairness or validity of evaluation by their own administrators, think how much less confident they must be about the judgment of state legislators (who have changed laws to comply with RttT) in this matter.