« The temptation tax | Main | Time is money »
Teacher Performance Pay: Experimental Evidence from India
In an impressive new paper, Karthik Muralidharan and Venkatesh Sundararaman provide evidence on the power of teacher incentives to increase learning. The paper is impressive for three reasons:
1) Evidence comes from a very large sample, 500 schools covering approximately 55,000 students, and treatment regimes and controls are randomly assigned to schools in a careful, stratified design.
2) An individual-incentive plan and a group-incentive plan are compared to a control group and to two types of unconditional extra-spending treatments (a block grant and hiring an extra teacher). Thus the authors can test not only whether an incentive plan works relative to no plan but also whether an incentive plan works relative to spending a similar amount of money on "improving schools."
3) The authors understand incentive design and they test for whether their incentive plan reduces learning on non-performance pay margins.
The results are as follows:
We find that the teacher performance pay program was highly effective in improving student learning. At the end of two years of the program, students in incentive schools performed significantly better than those in comparison schools by 0.28 and 0.16 standard deviations (SD) in math and language tests respectively....
We find no evidence of any adverse consequences as a result of the incentive programs. Incentive schools do significantly better on both mechanical components of the test (designed to reflect rote learning) and conceptual components of the test (designed to capture deeper understanding of the material),suggesting that the gains in test scores represent an actual increase in learning outcomes. Students in incentive schools do significantly better not only in math and language (for which there were incentives), but also in science and social studies (for which there were no incentives), suggesting positive spillover effects....
School-level group incentives and teacher-level individual incentives perform equally well in the first year of the program, but the individual incentive schools significantly outperformed the group incentive schools in the second year....
We find that performance-based bonus payments to teachers were a significantly more cost effective way of increasing student test scores compared to spending a similar amount of money unconditionally on additional schooling inputs.
Surprisingly, since absent teachers are a big problem in India, reduced teacher absenteeism per se does not appear to be the primary mechanism by which incentives improve learning. Instead the primary mechanism appears to be more intensive teaching, including more homework and classwork and better attention to weaker students, this greatly increases the relevance of these results to teaching in the developed world.
Addendum: See also Karthik's comments on the comments at 26.
Posted by Alex Tabarrok on September 30, 2009 at 07:39 AM in Economics, Education | Permalink
Comments
I am rectified
I argued just last week against workplace incentives,
using mental gymnastics, not statistics.
But definitive answers spout from the mouth of statistics. Left with only the minuscule doubt that we must reserve for any view on any subject, I will now talk and act as if teacher incentives are efficacious.
My and others mental gymnastics against teacher incentives was/is easy and unbridled. Statistical studies, and this large study in particular, are enormously difficult particularly in time. Muralidharan and Sundararaman paid the price to exact the truth.
While people might call for new affirmative studies, to outright argue against teacher performance pay should lead us not to question these statistical results but to severely question the intransigent in an impolite way.
I wonder how much of my thinking amounts to slosh, when a study like this forces my belief into a U-turn.
Posted by: Jameson Burt at Sep 30, 2009 8:47:25 AM
Hmmm.. comparing India to the U.S. might not be a very good idea. Its civil servants are not known for their intrinsic motivation.
But nevertheless, very interesting.
Posted by: JSK at Sep 30, 2009 9:20:41 AM
Well, no one can accuse Jameson Burt of confirmation bias. That has to be the most lucid and enjoyable narrative of Bayesian updating I've ever heard.
Posted by: Jeffrey Horn at Sep 30, 2009 10:00:53 AM
"Hmmm.. comparing India to the U.S. might not be a very good idea. Its civil servants are not known for their intrinsic motivation."
Uh huh. Racism alive and kicking!
Posted by: k at Sep 30, 2009 10:40:04 AM
Just as workers are less productive on an incentive program over time, I wonder if teachers would be as well. Sure the scores increased for the students, but 1. Who's to say the scores truly showed valid progress; 2. Will the heightened performance continue beyond 2 years?
A teacher motivated by incentives receives their motivation externally. Motivation in such a form is never sustainable. Over time, I would expect the motivation teachers receive from incentives would decrease over time.
I agree, this is an important study and it is great that incentives do truly show increased performance. Great. However, I truly believe there are much better solutions.
P.S.
Jameson, you are a master at saying absolutely nothing. Good job.
Posted by: David at Sep 30, 2009 10:45:32 AM
"Hmmm.. comparing India to the U.S. might not be a very good idea. Its civil servants are not known for their intrinsic motivation." Err... do you mean India or U.S.? If both then the comparison of course is relevant.
Posted by: Tuula at Sep 30, 2009 10:56:20 AM
This is not in any way comparable to anything in the United States, because the ability and desire of parents to help their children learn is very different in India compared to the United States. Put simply, Indians respect education a great deal more, and parents strive to help their children achieve.
In the United States, there is a significant correlation between student performance and parental involvement; indeed, the influence of the teacher on student performance is extremely minor.
Posted by: tim at Sep 30, 2009 11:25:41 AM
Do we have any idea of how flexible the education labor market is in India? For example, we might not expect as robust results in the US if it is difficult to toss teachers who just get higher salaries but continue doing poor jobs...?
Posted by: Admiral at Sep 30, 2009 11:35:59 AM
Its civil servants are not known for their intrinsic motivation.
Intrinsic motivation is the first thing that pops to mind when thinking about American public servants.
Posted by: Dbltap at Sep 30, 2009 11:56:19 AM
"0.28 and 0.16 standard deviations (SD)" ????
thats nothing statistically
Posted by: general at Sep 30, 2009 11:57:05 AM
A fantastic paper. Resistance to performance pay for teachers here in the UK remains extremely vocal, despite the fact that we supposedly already have a (very lax) performance pay system. Sound experimental evidence can only help the case for a more sensible remuneration system for teachers (he says optimistically...)
Posted by: Pockets at Sep 30, 2009 11:58:09 AM
How much of the improvement is due to teachers now having an incentive to help cheating in exams?
Posted by: skeptic at Sep 30, 2009 12:13:38 PM
@JSK:
And where exactly are civil servants noted for their intrinsic motivation?
Posted by: Anders at Sep 30, 2009 12:24:22 PM
U.S.? Let's read the second sentence of the abstract: "We present results from a randomized evaluation of a
teacher incentive program implemented across a large representative sample of government-run rural
primary schools in the Indian state of Andhra Pradesh."
Posted by: John B. Chilton at Sep 30, 2009 12:31:04 PM
what's the difference between the word effective and efficacious? i didnt know statistical studies were enormously difficult in particular to time. i appreciate you shedding light on that.
so if india and the us have different levels of intrinsic motivation, how would you:
1)determine that exactly
2)control for that in a model
Posted by: eh.nonymous at Sep 30, 2009 12:33:50 PM
The length of the test, two years, may counter what I'm about to write, but could another possible mechanism be that incentive schools attracted better teachers?
Posted by: Seth at Sep 30, 2009 12:54:17 PM
Read the paper:
The average rural primary school is quite small, with total enrollment of around 80 to 100
students and an average of 3 teachers across grades one through five.
One teacher typically teaches all subjects for a given grade (and often teaches more than one grade simultaneously). All regular teachers are employed by the state, and their salary is mostly determined by
experience and rank, with minor adjustments based on assignment location, but no component based on any measure of performance. The average salary of regular teachers is over Rs. 8,000/month and total compensation including benefits is close to Rs. 10,000/month (per capita income in AP is around Rs. 2,000/month; 1 US Dollar ≈ 48 Indian Rupees (Rs.)). Teacher
unions are strong and disciplinary action for non-performance is rare.
....
Regular civil-service teachers in AP are transferred once every three years on average.
While this could potentially bias our results if more teachers chose to stay in or tried to transfer
into the incentive schools, it is unlikely that this was the case since the treatments were
announced in August ’05, while the transfer process typically starts earlier in the year. There
was no statistically significant difference between any of the treatment groups in the extent of
teacher turnover or attrition, and the transfer rate was close to 33%, which is consistent with the
rotation of teachers once every 3 years (Table 1 – Panel B, rows 11-12). A more worrying
possibility was that additional teachers would try to transfer into the incentive schools in the
second year of the project. As part of the agreement between the Government of AP and the
Azim Premji Foundation, the Government agreed to minimize transfers into and out of the
sample schools for the duration of the study. The average teacher turnover in the second year
was only 5%, and once again, there was no significant difference in teacher transfer rates across
the various treatments (Table 1 – Panel B, rows 13 - 16).
Posted by: Arun at Sep 30, 2009 1:23:44 PM
America is the richest nation on earth; it's problems are basically nonexistent compared to those of India, particularly with respect to educational outcomes. Can we please just value this research solely on the basis that it told us something about how to perhaps make India richer? Or must we evaluate it with the US in mind, regardless of how much tired, uninformed partisan bickering our doing so stirs up and how relatively tiny the welfare consequences might be in any case (assuming that the results of the study are sufficiently externally generalizable)? Get some perspective dudes.
Posted by: curren caples at Sep 30, 2009 1:55:11 PM
It's a very interesting paper but of limited application to the U.S. since primary (elementary) schools are not the major problem.
Posted by: Paul Johnson at Sep 30, 2009 1:59:11 PM
this might prove something about marginal pay, but i doubt much more. the difference in life betweeen and indian teach making (completely uneducated guess) 100/month going to 200 a month would be vastly different than NYC teach making 3000/month to 3100/month
Posted by: farmer at Sep 30, 2009 2:06:33 PM
"the influence of the teacher on student performance is extremely minor" --tim
Really? I'm pretty sure that's the opposite of true.
"classroom teachers are the most important factor in achieving gains in student achievement"
http://www.archindy.org/NCEA/files/press/Teacher%20Impact%20on%20Student%20Proficiency%20and%20Growth%20NCEA.doc
"measures of teacher preparation and certification are by far the strongest correlates of student achievement in reading and mathematics"
http://epaa.asu.edu/epaa/v8n1/
"TFA teachers tend to have a positive effect on high school student test scores relative to non-TFA teachers"
http://www.urban.org/publications/411642.html
Posted by: Ben at Sep 30, 2009 2:44:25 PM
We find that the teacher performance pay program was highly effective in improving student learning. At the end of two years of the program, students in incentive schools performed significantly better than those in comparison schools by 0.28 and 0.16 standard deviations (SD) in math and language tests respectively....
In education, the standard for effectiveness is 0.25 standard deviations to be considered educationally significant. Below this threshold you're unlikely to get an observable improvement. So, the study shows barely educationally significant results in math and educationally insignicant results in language arts. In neither case, would I characterize the study as the authors do as being "highly effective in improving student learning."
Posted by: kderosa at Sep 30, 2009 3:47:30 PM
Paul Johnson says:
"It's a very interesting paper but of limited application to the U.S. since primary (elementary) schools are not the major problem."
Really? High school teachers certainly say that the kids they get from middle schools are often not up to grade level. I suspect that middle school teachers would say that they also get many unready students.
(I'm not saying that the paper necessarily is highly applicable to the US, just that this rationale for it not being highly applicable is off-base.)
Max
Posted by: Maximum Liberty at Sep 30, 2009 5:28:19 PM
Arun - Thanks.
Posted by: Seth at Sep 30, 2009 5:40:54 PM
Arun - I'm not buying their logic on transfer rates of teachers. First, the rates don't say say who transferred where. Perhaps the better teachers, transferred to the reward schools and other teachers transferred out, not wanting to face the extra accountability.
Also, the 33% transfer rate in the first year, which is "consistent with the
rotation of teachers once every 3 years" while the 5% turnover rate in the second year raises an eyebrow for me. Why isn't the transfer rate in the second year not consistent with the "rotation every 3 years"?
Posted by: Seth at Sep 30, 2009 6:07:59 PM
