« Gender and Insurance | Main | The Base »

Cash for Test Scores

26 Jun 2008 01:41 pm

A Prince George's Country merit pay plan for teachers seems to have gained support among the local union, which will certainly make the experiment noteworthy. I'll be interested to see how it turns out. You can mark me down as a skeptic, though, since it seems very likely to me that linking teacher compensation to test scores this directly mostly creates incentives for teachers to help their students cheat on tests.

However, given that this is a pretty obvious idea and we have a whole lot of school districts in this country, I think it's essentially inevitable that this will be tried out in some places and we'll see what the results are. I think, however, that there are probably more promising ways of caching out the idea of "merit pay" that would focus more on the idea of giving administrators flexibility with teacher salaries and then holding them accountable for getting results.

Share This

Comments (45)

Why not pay the students directly?

Here's a crazy thought: since the primary key to academic success is probably the motivation of the student, why don't we pay them to do well? We seem to be building these systems where everyone is given incentives (whether carrot or stick) besides the kids.

This must have been tried somewhere, at some point, right?

If I was a teacher in this district, I would focus on sabotaging other teachers more than helping my students cheat. No one would see it coming!


Why not try the direct approach and pay STUDENTS for good test scores? Imagine how the social dynamics of the school cafeteria would change if the geeks and nerds had money, all of a sudden :-)

-- TP

Why not try the direct approach and pay STUDENTS for good test scores? Imagine how the social dynamics of the school cafeteria would change if the geeks and nerds had money, all of a sudden :-)

-- TP

Posted by Tony P. | June 26, 2008 2:05 PM

Bullies would be taking lunch money from middle and high schoolers in addition to elementary schoolers?

I dissent from the view that we should pay the students.

I also disagree that the primary effect would be teachers helping their kids cheat. I think the primary effect would be teachers going to great lengths to cull the below average students from their classrooms. Transfers and expulsions would become very political.

Posted this when the discussion came up at The American Prospect, but seems relevant again here:

I hope we get merit pay that *isn't* based on test scores. A couple of friends of mine are public high school and junior high teachers here in New York, and had some very interesting things to say about merit pay in general.

They're basically all for it. As they say, it's very easy to be a crappy teacher, and a lot more work to be a good teacher. But there's no pay incentive to be good rather than crappy, and much as we'd like human idealism to fill the gap, it's easy to get burnt out and end up just doing the minimal work you need to get by when there's no personal reward for doing better.

The problem is that merit pay based on test scores creates all kinds of bad incentives, particularly in terms of making it really undesirable to teach low-scoring kids who need help most. So they support a plan that's apparently being tried in Denver, where merit pay is accorded as a series of bonuses based on a broad set of actions---start an after school program, you get $X, raise kids' scores by a certain amount (not to an absolute level, but by a percentage), you get $Y, teach this understaffed subject, you get $Z, and so on.

That seems like a good way to get the incentives of merit pay, without the negative consequences. And it's the hope for that sort of program that has my friends bucking their union to support Obama (and working quietly to change their union leadership, which they have a lot of other problems with).

I think the primary effect would be teachers going to great lengths to cull the below average students from their classrooms.

Isn't this a pretty well known phenomenon from the NLCB practice? That there is a real effort by the schools to dispose themselves of their worst students?

I dissent from the view that we should pay the students.

I had not thought about this until reading the post, and clearly the thought struck others. Why would this be such a bad idea?

I should add a caveat---that was written back when the local teacher's union was backing HRC, which is why I say "bucking their union." Now, of course, their local is for Obama.

Yeah, if a pretty tall and strong wall is not erected between the testing regime and the teachers, with very frequent test question changes, among other measures, and test grading completely out of the teachers' hands, there will be a fair number of teachers who will cheat like hell to get more cash. This is no particular criticism of teachers; I'd expect the same behavior from any large group of people. There's a reason why casino owners watch the blackjack dealers like hawks, and banks count the tellers' tills regularly.

And even if they don't literally "cheat" by feeding the kids actual answers to actual question, the time they will waste by making a curriculum solely out of the tiny fraction of stuff that can be tested on standardized tests is awful.

Look at a standardized test sometime. Only certain kinds of knowledge are tested on it and those kinds are often the most useless. (E.g., the NY State Regents exams in math give a lot of credit for questions that test solely the student's ability to memorize some definitions; they do not give nearly enough credit for the ability to solve problems, since that's hard to test in the time and manner allowed.) So expect your kids to be learning lots of pointless nonsense in order to get their teacher a raise.

Brad L., I think you are correct that it would be a mistake to dismiss the notion out of hand. It may even be worthwhle for states to offer substantial tutition discounts from state universities and colleges to students who perform well in junior high and high school. It may be cheaper to motivate students and parents to work hard towards achievement in junior high and high school than it is to not offer such incentives while collecting more tuition at the universities and colleges.

The cheating fear is just silly. It's an incredibly simple matter to keep the specific test materials confidential until the day of the test and have someone other than student's teacher proctor the test. With those percautions in place, the easiest way to raise the students test scores is to actually teach them the material.

It's worth noting that bonuses are applied in a far less structured manner in the private sector without substantial problems. Unless teachers as a whole are grossly less professional than engineers, account managers, and so on, widespread misconduct in order to maximize bonuses is unlikely. It's important to make the tests represent the important aspects of the cirriculum well so that teachers focus on the right things, but I can't imagine testing-linked bonuses leading to substantial misconduct.

MattXIV, bonuses are frequently a problem in the private sector. When metrics are used to determine compensation, cooking the metrics is sure to follow, absent competent controls. Yes, it may seem simple to avoid the issue, but the frequency with which simple measures are not taken is surprising. That doesn't mean that the whole idea is worthless, of course, but as with everything, the devil is in the details.

The cheating fear is just silly. It's an incredibly simple matter to keep the specific test materials confidential until the day of the test and have someone other than student's teacher proctor the test. With those percautions in place, the easiest way to raise the students test scores is to actually teach them the material.

It's worth noting that bonuses are applied in a far less structured manner in the private sector without substantial problems. Unless teachers as a whole are grossly less professional than engineers, account managers, and so on, widespread misconduct in order to maximize bonuses is unlikely. It's important to make the tests represent the important aspects of the cirriculum well so that teachers focus on the right things, but I can't imagine testing-linked bonuses leading to substantial misconduct.

I had not thought about this until reading the post, and clearly the thought struck others. Why would this be such a bad idea?

I don't think it's the worst idea I've ever heard, but a few objections that spring to mind:

The Chris Rock Objection. You're supposed to be doing this anyway. We the taxpayers go to massive (though, I admit, underfunded) expense to provide a tuition-less education for these kids, and then we're expected to pay them for the privilege? Should I also pay my son for eating the vegetables I bought for him? It seems like a bad paradigm.

The Milton Friedman Objection. There are already massive incentives for doing well in school. That's the path most of us take to prosperity or to careers we find truly satisfying. We should focus on making those incredible incentives more apparent to kids, but it's wasteful and inefficient to layer new incentives on top of that. Further, it creates a culture of children who expect instant rewards from their efforts. And from a purely pragmatic standpoint, it's probably fiscally unsustainable to fund this sort of thing from K-12 . . . to produce the incentives you're talking about would be really expensive.

The Morgan Freeman as Principal Joe Clark Objection. It's probably not a great idea to build a significant cash economy within the walls of a school. That could lead to all sorts of mischief.

Only 25 percent of this plan is based on improving individual scores, another 25 percent is based on improvement of campus scores, and the remaining 50 percent is unrelated to scores, so to argue that this plan is entirely about paying individual teachers for their test scores is rather disingenuous. An entirely score-based plan is used in Houston, having started last year. See here for the details. Not really sure if it had any effect on teacher behavior. (However, note that Houston DOES pay stipends to teachers in understaffed fields--it's just not part of this program).

Of course, test scores aren't everything. There are tons of problems with using test scores--maintaining test standards in the face of overwhelming pressure, different ways of measuring what teachers should achieve, differences in incoming classes (i.e. how well students are prepared). Those who say this would provide incentives for teachers to teach specific classes and avoid others are correct. Not to mention that there are other things than test scores. The idea of giving teachers money for after-school programs, more training and innovation is better, but of course even harder to measure.

As for paying for scores... well, Bloomberg proposed this in New York and I don't think it was adopted. I wrote something else, but see what southpaw wrote.

Will Allen:
It may even be worthwhle for states to offer substantial tutition discounts from state universities and colleges to students who perform well in junior high and high school.

...for who? everyone or just low SES students? If the former, it's serving a different policy objective than improving K-12 performance. Even if just directed at low SES students, the ones who don't think of college as an option (i.e. low scorers) derive zero utility from college tuition discounts. I do think low SES students who achieve should have more help, just don't think it's necessarily the best vehicle for helping low performers.

from the post:
giving administrators flexibility with teacher salaries and then holding them accountable for getting results.

Hmmm. Within certain parameters, I could see this working. However, it would require administrators with much more longevity in their positions than I've seen, to ensure stability. That means attracting and retaining competent administrators. A system like this could cause chaos if half the administrators are incompetent and arbitrary--which is quite possible in some districts.

At the end of the day, teachers and administrators aren't the only, or the most important, factors in a student's life, and it's important to remember that and order policy preferences accordingly.

The problem is not cheating. Teachers will simply teach to the test.

The problem is with the assessment mechanisms being used. I ran a literacy program that used standardized tests and highly structured reading programs benchmarked to those assessments for about six years. These assessments can only measure the mechanics of reading de-contextualized and abstract text. They cannot measure the higher cognitive skills we associate with 'literacy'... synthesis, reason, etc.

Such policies may produce a great deal of 'hard data' and may make for easy mechanisms to dole out 'merit pay'. However, it provides the kind of education that is the polar opposite of what is required to actively participate in a republican system of governance.

Cheers!

I don't think it's the worst idea I've ever heard, but a few objections that spring to mind:

Ok. I'll take a stab at this. Please bear in mind that I am still forming an opinion, not representing a well-honed view. If I sound like I'm muddling through, it's probably because I am.

You're supposed to be doing this anyway.

It's a funny joke about tipping cabbies (um, or whatever the original context was... sorry! Totally unrelated cred-finder: I saw Wanda Sykes in a tiny club about 6-7 years ago, and she was incredibly funny. She doesn't get nearly enough credit as one of Rock's writers.)

But for the most part, this really applies to people that are already doing their jobs. Kids in school are compelled to be their, which is different. And, they aren't adults, so their sense of purpose and responsibility might not be so well-formed.

There are already massive incentives for doing well in school. That's the path most of us take to prosperity or to careers we find truly satisfying.

For the group that current incentives are failing, they are failing miserably. The incentives in place are long term (college, etc), and may seem quite out of reach to many. Having short-term incentives to supplement this seems like a practical solution.

It's probably not a great idea to build a significant cash economy within the walls of a school.

This is an interesting one. I'm not sure how true this is (it seems to me there are plenty of cash economies already taking place). But there must be incentives that are more immediate than "college someday" that would work.

One more . . .

The Dirty Fucking Hippie Objection. This society is already too focused on its money culture. To teach kids that everything they do in school is and ought to be for monetary gain will provide that money culture with a huge steroid injection. We want kids to things other than corporate law and investment banking. We want them to go into low paying public interest work or academics or artistic pursuits. We want them to be willing to sacrifice to become great writers or aid workers or firefighters or soldiers or teachers. To teach kids, in the public schools, that every time they do something right they're going to get paid for it seems deeply destructive of that end. It associates merit with immediate profit, and neither the real world nor our ideal world is going to work out that way.

Sorry for the sloppy writing in the last two posts; I'm muddling through this myself.

Kids in school are compelled to be their

And, of course, unintended irony in our discussion about learnin'. Kids in school are compelled to be there. Of course, on this blog, that hardly counts as an oopsy.

Will Allen:
Immediate rewards are far more effective than distant ones. Imagine a pinball machine that rewarded your 'passing score' with a certificate for a free game in 10 years, rather than a free game right now.

What I would like to do is get some quirky philanthropist to start a nationwide chain of 'Quiz Parlors', akin to the pinball arcades of old. Kids would come in and take tests, with cash prizes. No pass/fail criteria -- just so many dollars per correct answer. More dollars for harder questions.

The philanthropist could think of it as a sort of 'retail scholarship program': a kid could (in principle, at least) earn money for college a few bucks at a time. I say this approach would motivate kids more than the promise of a wholesale college scholarship in the (to him) distant future.

An important feature of this crazy proposal is the separation between the teaching function and the testing function. The philanthropist would decide what the quiz questions should be, and what the right answers are. He would not have to tell anybody in advance; hence 'teaching to the test' would be hard to do.

The reward to teachers would be higher motivation among their students. Easier work, not higher pay. I think teachers deserve higher pay, mind you, but that's a different discussion.

-- TP

This society is already too focused on its money culture. To teach kids that everything they do in school is and ought to be for monetary gain will provide that money culture with a huge steroid injection.

This is tricky: the big, unspoken incentive for most high-schoolers is long term success. In our culture, this largely means money (do well in HS, do well in college, get a well-paying job), but it can also be defined as other success: getting interesting jobs (or widely respected jobs) that don't pay as well. But that first big incentive -- money -- is there from the start. We're just changing the time horizon: is it long term, or shorter term?

OTOH, I can imagine an incentive system that is partially personal goods (cash, or something different), and partially community goods. Better scores means better facilities (library, gym, lab)? Better field trips?

I really like the idea of putting some skin in the game for students who otherwise don't feel motivated. I'm not sure what this would look like, I admit, but the underlying principle seems really enticing.

A Prince George's Country...

So, which country is Prince George's?

This is a FAR better idea if the pay is for the average improvement in individual students' scores rather than looking at class mean improvement. This is one of the key flaws in NCLB. Cross-sectional testing is a lazy approach and leads to all sorts of problems. longitudinal testing, where you track individual student performance and changes, is far superior. This is similar to longitudinal versus cross-sectional epidemiological studies, and are why longitudinal studies are the gold standard.

An additional reason for doing longitudinal assessments is that they are far more useful for individual student evaluation and guidance.

Making the transition to longitudinal assessment is difficult. I believe that NY City is in the process. If NCLB had gone that route it would have been much more valuable.

The way to grade teachers is by using the test scores of their students that they take NEXT YEAR.

Teacher A and Teacher B each teach 30 4th grade students. The average score in Teacher A's class is 75 and the average for Teacher B's class is 55.

Then, next year, the average scores in 5th grade for the kids from Teacher A's 4th grade class is 75 and the average for Teacher B's 4th grade class is 60.

This means that Teacher B did a better job teaching last year's 4th graders than Teacher A did.

You can't teach to a test that the kids will take next year because they aren't ready to learn the material yet. It greatly reduces cheating. The teachers giving the test are not judged on the outcome of the tests.

Good teachers will teach kids and that will carry over into the next year. It is not the most direct way to test teachers but it is accurate and effective.

The problem with paying students for good grades is that it will be a regressive benefit: The richer the family, the more likely their kids will get paid.

Interesting idea, but there are a couple of problems with that one. First, how would you pay 12th-grade teachers?

Second, suppose you have two equally good fifth-grade teachers; but one great and one terrible sixth-grade teacher. The kids in the great 6th-grade classroom are interested, motivated, and do very well; the kids in the other classroom are bored silly and start flunking. In that scenario, the two fifth-grade teachers wouldn't be paid the same, when it really wasn't their fault that their kids screwed up (or excelled) in the next year.

Like I said, it's an interesting idea; and maybe some portion of their pay could be based on future performance. But I don't think it should completely determine pay.

Obviously, there is a gigantic conflict of interest in current K-12 testing. No Child Left Behind tells the states to make up their own tests, administer the tests, grade the tests, then report back to Washington on whether the test scores have gone up enough to qualify for federal bucks. That's why Mississippi, that intellectual powerhouse, has, officially, the highest percentage of proficient readers in the country!

What America needs are independent K-12 testing agencies. For college admissions testing, we already have two: ACT and ETS/College Board. You'll notice college admissions test scores don't suddenly zoom upwards from one year to the next the way they often do on state-created tests. The independent agencies don't have an incentive to cheat.

"That's why Mississippi, that intellectual powerhouse, has, officially, the highest percentage of proficient readers in the country!"

Wow!

But why am I not convinced this is the answer: "independent K-12 testing agencies"? Maybe because I haven't thought the models cited, ACT and ETS, have done such a good job themselves. And while they may not "zoom," they get "re-centered" and their errors result in faulty scores more often that is entirely comforting.

The next fundamental step needed besides independent K-12 testing agencies is that we need a baseline measurement for every student in America of their intelligence. That way we can compare their school achievement scores to their IQ to see how much value the schools are adding.

For example, in Los Angeles county, the non-exclusive public high school with the highest average SAT score is San Marino at 1230. So, the staff and teachers of San Marino must be doing a bang-up job, right? Actually, nobody knows. Students tend to be the scions of Hong Kong millionaires, so anybody not stupefied by political correctness would expect them to do well because the average IQ of the students is so high.

So, what we need is to have each student tested by an independent agency for IQ when he or she starts at a school. The figures would be kept encoded in a national database and the school would be graded on how much achievement it elicits from its students relative to their IQ.

Schools could use this information as well to specialize for different classes of students -- they could advertise to parents that they are a good school at adding value for students with two digit IQs or for students with IQs over 115 or whatever.

As for the objection that teachers will only teach to the test, they already are. I taught in Houston when the perofrmance pay started, and teachers were already required to teach entirely to the test. Curriculum standards were designed to insure it, and any teacher over the established curriculum would find themselves the subject of daily observatiosn from adminstrators to make sure they got back on track.

ETS and ACT aren't perfect, but they play a large role in the world-beating prestige of American universities. On that Chinese list of top universities, 17 of the top 20 are American. That's in part because American universities, despite all their politically correct rhetoric, are viciously elitist about test scores, such as the GRE. And the GRE does a good job of figuring how who the best potential physicists or whatever would be.

In contrast, continental European universities fell for all that 1968 leftist egalitarianism and largely shifted over to open admissions. That's a major reason why you almost never hear about the famous old German or French universities anymore like Gottigen or Sorbonne.

http://isteve.blogspot.com/2008/06/american-education-paradox.html

And while they may not "zoom," they get "re-centered" and their errors result in faulty scores more often that is entirely comforting.

This is true perhaps on an individual level: the test doesn't accurately reflect a particular students level of learning. On a broader (statewide or even schoolwide) level, it's probably a better metric, and when used for judging aggregate teaching outcomes (rather than individual student placement), this wouldn't be such a big problem.

There still might be "teaching the test" type problems -- we generally thing of a good education as more than simply the fundaments, but if we only reward the basics (or even advanced, but very specific, techniques) then that is what the teaching will emphasize. My recollection of the SATs is that it was very algebra/geometry/trig focused in math, and quite vocabulary focused in english. Science and history are, of course, left off.

This raises the real question: how broad do we want our education, and how important is it?

Here's that Chinese list of top universities. The University of Tokyo is #20, and above that are Oxford and Cambridge and 17 American universities.

http://en.wikipedia.org/wiki/Academic_Ranking_of_World_Universities

"Science and history are, of course, left off."

I took the SAT's prior to graduating high school in 1999. At least back then, while science etc. were not included in the main SAT I, there are a bunch of specific subject area tests in the SAT II. (AP tests do the same). While the SAT I's were used to get into college, the SAT II's were generally used to see if you tested out of required college courses.

As for the objection that teachers will only teach to the test, they already are.

Hence the objection to the approach, no?

The basic point is that you can learn a lot about the basics of fixing education by comparing the differences between college and K-12. For example, we have two independent national testing agencies for college and grad school admissions, so we don't have the massive conflict of interest that undermines K-12 testing.

One problem, though, is that American universities put out a lot of politically propaganda to mask their Social Darwinian IQ elitism. Back in the 1960s, the only famous college that took leftism seriously was poor CCNY, which went to open admissions, and immediately collapsed in prestige. All the other colleges merely instituted affirmative action to buy off smart minorities, and upped the ante on their IQ elitism in admissions, while propagandizing ever harder on the political correctness front to cover up their tracks.

This is a problem because the Harvards and Yales are so enormously prestigious that the poor saps in the K-12 business fall for what they say, not what they do. Thus, the reigning ideology of American K-12 is happy-clappy egalitarianism, with predictably ineffectual results.

In contrast, German universities don't have much prestige because they shifted over to open admissions, so their leftist rhetoric hasn't wrecked Germany's K-12 system of tracking students into academic and vocational/apprentice routes based on their academic capabilities.

http://www.vdare.com/sailer/080622_paradox.htm

Wow. Just when I thought Steve Sailer had actually made a reasonable comment -- the thing about the need for independent testing agencies -- he follows it up with a suggestion that we have a government database with every American kid's IQ. Thank you, Steve. If you had left it at just the reasonable, non-racist comment, my faith in the order of the universe would have been shaken.

As far as "teaching to the test," that's the whole point. If the test is measuing the right things, then teachers should be teaching to the test. They can add their own flourishes only if the kids know the material that's on the test. Again, that's assuming the tests are measuring the right things.

One reason that american universities are so good is that they are expensive. In other words, we put a lot of money into higher ed and it pays off. Its not just the top schools; quality education runs the gamut.

Among the myriad problems with k-12 education is lack of financing. People scream about paying 10k per student in a k-12 system and you hear fewer squawks about the 40k per year in tuition at private universities and comparable costs (although not tuition) at public universities.

Neil's suggestion, testing the next year's performance, is a pared down form of longitudinal testing (you are tracking the class of students from current year to next year). It has merits, but, compared to thorough longitudinal testing loses a lot.

"Wow. Just when I thought Steve Sailer had actually made a reasonable comment -- the thing about the need for independent testing agencies -- he follows it up with a suggestion that we have a government database with every American kid's IQ. Thank you, Steve."

Every state currently has one or more databases with public school students' achievement tests stored by the name of the student. The federal government knows all your income and its sources, whom you donated money to, etc. etc. Private companies have huge files on your credit history. The College Board has databases on tens of millions of people's test scores. The Pentagon has a database of tens of millions of Americans IQ scores from the AFQT military enlistment test.

So, why are those A-OK, but not having a system where we can actually evaluate how good a job each school is doing?

As far as merit pay goes: what he said.

I would also like to point out that the biggest problem facing the teaching profession is retention (about half of new teachers are gone within 5 years) and one of the biggest reasons new teachers leave the profession is lack of support (more than lack of money). How much support are new teachers going to get from their experienced colleagues if they're all in competition for Test Bucks?

Why is everybody that's posting on here thinking so hard about this topic and trying to win the unspoken competition of "best post?" Because we are VOLUNTARILY involved, engaged, and interested in the subject.
Last year I had 126 kids in six classes. The basic idea of each class, or so I still hope, was that each kid would be involved in thinking about worthwhile things and expressing themselves more clearly.
Real change in education will be remarkably difficult because it's going to have to come from the bottom up (I've heard that expression somewhere before - I just can't remember where): better parents, better teachers, and a better culture.
I've been teaching for twelve years and I like it, basically, because I like young people and I like a daily challenge. I think it's a hard job just to be adequate. But if the day comes when I absolutely, positively have to look at 126 people simply as potential test scores (and that's where we seem to be going) you can include me out.


Comments closed July 10, 2008.

Copyright © 2008 by The Atlantic Monthly Group. All rights reserved.