« Stupid Box Tricks for Intellectuals | Main | Questions that are rarely asked, a continuing series »
Rating RateMyProfessors.com
You’ve heard the reasons why professors don’t trust RateMyProfessors.com, the Web site to which students flock. Students who don’t do the work have equal say with those who do. The best way to get good ratings is to be relatively easy on grades, good looking or both, and so forth. But what if the much derided Web site’s rankings have a high correlation with markers that are more widely accepted as measures of faculty performance? Last year, a scholarly study found a high correlation between RateMyProfessors.com and a university’s own system of student evaluations. Now, a new study is finding a high correlation between RateMyProfessors and a student evaluation system used nationally.
Strike another victory for Web 2.0. Here is more.
Posted by Tyler Cowen on April 25, 2008 at 07:18 AM in Education | Permalink
Comments
...and another defeat for student evaluation of teaching?
Posted by: Ed Lopez at Apr 25, 2008 7:56:01 AM
Quick look at my alma mater suggests that the observed hot/easy/good correlation is much tighter in undergraduate programs than in graduate programs.
Posted by: Andrew Edwards at Apr 25, 2008 8:00:55 AM
"For his part, Sonntag acknowledged that some RateMyProfessors.com reviews are “so mean-spirited” that they aren’t worth anyone’s time. "
I think that this is the crux of the issue; professors don’t enjoy being insulted anonymously with no chance of correcting the statement. The main thing this study seems to uncover is that the same biases that ruin RateMyProfessors.com credibility ruin university-sponsored student evaluations as well. All RMP adds in are nasty and unmoderated statements. RateMyStudents.com is already taken, unfortunately, because I'd love to have a site where professors could offer the same evaluations of their students. Employers could probably make great use of the site, and the value of academic signaling would be increased, benefiting students who make an effort. Of course, some students would be outraged, because how could anyone dare say something bad about THEM!
Posted by: Aaron at Apr 25, 2008 8:30:43 AM
Um, don't professors already have an outlet for rating students?
Posted by: Josh at Apr 25, 2008 8:53:23 AM
I take great joy in the discovery there’s a high correlation between results of student evaluations and RateMyProfessor.com. Wow, Voice of the Customer comes to higher education, now if only someone will listen and respond accordingly.
I write the above as a 53 year old who last year returned to graduate school in the School of Public Policy at GMU; I attended a large public university as an undergraduate and a small private college for an MBA. I also have a daughter finishing her junior year at a small public liberal arts institution and so also have a parent’s perspective
My experience as an undergraduate student in the 1970s, graduate student in the 1980s and currently, alumni and current parent is that all higher education really cares about is accessing my wallet. Customer satisfaction (with customers being defined as students and parents) is an unknown philosophy, with perhaps the exception of dorm space, food service offerings and extracurricular offerings. How teachers actually teach is never addressed, except as a question to the tour guide on your typical undergraduate tour.
I love the feedback on RateMyProfessor; maybe the classifications could use some work and refinement but it’s a great start and there’s great performance information in there.
I can understand academicians don’t like it but it’s time for the academy to move out of the middle ages and compete with performance in the classroom with customer satisfaction one of the metrics.
Posted by: Dave Richardson at Apr 25, 2008 9:51:35 AM
The best way to get good ratings is to be relatively easy on grades, good looking or both, and so forth.
Last year, a scholarly study found a high correlation between RateMyProfessors.com and a university’s own system of student evaluations.
There is absolutely no conflict between these two statements.
Posted by: Anthony at Apr 25, 2008 10:08:41 AM
Don't student evaluations suffer from the same problem as the web site? Students who don't do the work can just as easily fill out the evaluation as log onto the site.
I'm finishing up my undergraduate studies, and have always tried to leave well-reasoned feedback on RateMyProfessor.com. However, there is a lot of "I got a bad grade and will therefore complain".
Posted by: Bo at Apr 25, 2008 10:14:45 AM
Bo wrote: "Don't student evaluations suffer from the same problem as the web site? Students who don't do the work can just as easily fill out the evaluation as log onto the site."
Exactly correct, however the good thing about RateMyProfessor.com is that it is not actually used to evaluate faculty performances by university administrations that are setting faculty salaries.
Grade inflation is the inevitable consequence of the use of student evaluations of faculty by university administrators. Grade inflation has gotten so bad that some universities have instituted grade "caps" of various kinds. At the university where I teach, student evaluations are used as the primary indicator of faculty performance. While I have no problem with them being used after adjustments are made for obvious control variables (such as students' SAT scores, to proxy ability, student attendance, to proxy their due diligence, and student grade in the class, to proxy their performance in the class), I think that using student evaluations as an unadjusted measure is simply contrary to what any student passing undergraduate econometrics understands: leaving out all control variables is not good practice.
Posted by: indiana jim at Apr 25, 2008 10:30:00 AM
Good news for Cowen and bad for Tabarrok.
I'd trust ratemyprofessors and especially pickaprof (which includes grading history for some schools) a lot more than the officially sanctioned feedback. Students don't trust the anonymity of the latter and don't want to rattle the cages of profs they might see again even if they do trust it. Rate and pick are intended for other students and not meant to be fed back into some bureaucracy where they'll do nothing.
Sure, they're gonna skew low: malcontents are more likely to post than champions. But malcontents often have good points.
Posted by: burger flipper at Apr 25, 2008 10:43:22 AM
I read my student evaluations through the typical means, but I refuse to look myself up on ratemyprofessor. After reading comments for other faculty I know to be good or great, students slam profs for fashion or say nasty things unrelated to the class.
Why subject my self to this.
Posted by: seth at Apr 25, 2008 10:48:54 AM
Student evaluations are nonsense.
At my university, we have group final exams. There are a few (hard) professors who get terrible evaluations. But on the group final (all sections of calculus X take identical finals, except for changed numbers/orders), their students tend to score 5-10% higher than the average.
One of them explained it to me: "my students hate me because they only understand 2/3 of what I say. But I say 2x as much as anyone else, so they come out ahead. They just don't know or like that."
Posted by: mathgeek at Apr 25, 2008 10:49:28 AM
What's up with the .4 hot/easy correlation?
Posted by: Douglas Knight at Apr 25, 2008 10:53:22 AM
I've read my own ratemyprofessor rankings and even the bad ones are pretty accurate. I've read those of my friends and enemies; also pretty accurate. The problem with ratemyprofessor is that the scale is inverted; easy professors get smiley faces hard ones not. This certainly drives off the lazier students, and depresses course enrolments in a potentially troubling way but no more so than gossip in the dorms. I also think this does not reflect the values of all students and even can act as a signal to those students who want to learn.
The problem is not with student evaluations it is with the way they are used by faculty. I have seen more than one colleague on a tenure committee pick a very witty turn of phrase and use it as the sole basis for a devastating attack on junior faculty member's teaching ability. In my own experience I would certainly like to throttle the single student who wrote that my philosophy of science class was dismissive of sociology. Not because sociology should not be dismissed but because two of the five people on my tenure review committee are from the sociology department.
Posted by: Kevin G at Apr 25, 2008 11:16:38 AM
Here are bits of five of the comments that are posted for me:
1) "AVOID AT ALL COSTS!!"
2) "The best grade on our first test was a 61%"
3) "His expectations are unreasonable because his philosophy is you need to get beat up in his class in order to not get beat up in the world. This guy is ****ed up. Do not take a class with him as your prof. I'm forewarning you. I'm an "A" student and got a C- in one of his classes."
4)". . . he's a good prof. Stick with it, read the book, DO THE HOMEWORK IF YOU WANT A GOOD GRADE ON THE TEST, and go to class every day, and you'll be fine."
5) "This class is cake as long as you go every day."
From the above I conclude what? Well, it seems that students who attend and do homework consider me a "good prof." whose class is "cake as long as you go every day." On the other hand, for those who don't want preparation for the world, I am "****ed up." So be it.
Posted by: indiana jim at Apr 25, 2008 11:31:07 AM
As an undergraduate student, I don't trust a damn thing of what anyone else says. Mostly because what everyone else says is highly variable, and young students tend to be highly emotional while criticizing/complimenting their professors.
I'd say a Rate My Professor rating isn't really useful until the student has had some time to digest what really happened. I absolutely hated my marketing teacher at the time I took the class, but I learned later on that he really helped us learn about the world
Posted by: Robert Olson at Apr 25, 2008 12:45:37 PM
right out of high school in my first undergrad program, I had no clue who were the good profs. in my late 20s in a 2nd undergrad program and then a grad program, I took professors not classes, and picked them using a combination of recommendations from 1) other students (the ones who asked intelligent questions in class and who were usually prepared) and 2) professors I had taken previously who I liked and who I believed knew their stuff.
I had a tax prof who few of my classmates liked, but he came highly recommended by a professor I respected. That tax professor was OUTSTANDING - he not only knew the code, he could talk about the leg history/policy behind it. He was one of the toughest profs I ever had, but he really brought the tax code alive for me.
Want to know who is is good in the classroom? Ask the students who you know are hard working and previous professors you respect.
Posted by: chug at Apr 25, 2008 12:45:42 PM
ratemyprofessor.com is a good website because professors aren't interested in real feedback from students, so don't listen or blow off criticism with the thought "he's MY student, what does HE know about teaching this stuff?" Well students may not have first hand knowledge of teaching, but they have first hand knowledge of learning. Most professors make their subjects unnecessarily hard because they don't want to see themselves as soft professors. This site isn't for the bad professors, but for students interested in knowing who the bad professors are.
I am in graduate school right now. I'm not sure how much of a difference there is between bad professors and hard professors. You'll get comments like this, from above:
"One of them explained it to me: 'my students hate me because they only understand 2/3 of what I say. But I say 2x as much as anyone else, so they come out ahead. They just don't know or like that.'"
This is pathetic and clearly shows that many professors DO NOT take pedagogic concerns into account, then rationalize their own foolishness. Teachers that do not know how to properly balance their own courses to make sure the amount being taught in the class isn't so overwhelming that the highest grade isn't 61% are just as bad as teachers that cannot explain things clearly. Making students unable to understand the material because you've decided to present WAY to much material, or my favorite, the same material "from alternative perspectives" (this one is especially idiotic since the ability to appreciate alternative perspectives DEPENDS on understanding at least one perspective all ready) is just as bad as not being able to clearly explain the material to your class.
Any student truly interested about a professor will use any means available to find out about a professor. Without fail, the worst rated professors really do turn out to be the worst professors. And if I have any doubt, I ask my fellow mathematicians, who will tell you straight up if the guy is bad, hard, easy, good to learn from, etc.
Again, websites like this seem to upset faculty and the administration so much because these sites are uncontrollable and the truth of the idiocy of higher education gets exposed. This is something they do not find amusing. Welcome to the information age, fellas.
Posted by: Ken at Apr 25, 2008 12:58:04 PM
I for one use ratemyprofessor all the time in picking classes. If a professor receives a lot of bad ratings from lazy students, there's a good chance he or she is a fine teacher.
Posted by: Michael at Apr 25, 2008 1:02:22 PM
Indiana Jim:
Are all of those comments from the same course? There is one professor at the university I attend with RMP reviews that average "Horrible course, horrible professor" for a 100 level gen-ed course and "Hard as hell, but you'll learn a lot" for a 300 level course.
Posted by: Thomas at Apr 25, 2008 2:07:30 PM
Thomas,
Comments 1, 2, 3, and 4 are for Intermediate Price Theory, and comment 5 is for Econ 116 (a beginning survey class).
Posted by: indiana jim at Apr 25, 2008 5:10:53 PM
Ken wrote: "websites like this seem to upset faculty and the administration so much because these sites are uncontrollable and the truth of the idiocy of higher education gets exposed. This is something they do not find amusing. Welcome to the information age, fellas."
Well Ken, I'm not upset in the slightest. In fact, I am happy for RateMyProfessor.com. When students post comments like this guy is "****ed up" they just make it clear that ad homenim is not below them. When the same student argues that I'm an "A" student and got a C- in one of his classes" this indicates that he/she either: 1) does not understand that by getting a C- he/she is no longer an "A" student (if he/she means a straight A student); or 2) he/she believes that because he/she has gotten A's in the past that he/she deserves them henceforth regardless of performance, field of study, input of effort, etc. Being hectored in such a way is a clear signal to thoughtful readers of the dubious intellectual integrity of the post.
No, I think that the posts can be highly revealing in a way that was in no way the intention of the person posting. This has the ring of the invisible hand.
Posted by: indiana jim at Apr 25, 2008 5:31:21 PM
I am sure that there is such a correlation. However, for professors who do not fit into a normal distribution,
small sample bias can have them way off. I will simply note that my overall ranking number is below the equivalent
number (we use the same scale) that I have received in any of the well over 100 classes I have taught, in which
all the students in the class were sampled. I actually called for the official evaluations to be made public
because of this problem, but needless to say that has not happened. So, as it is the 1 out of 20 students who
really, seriously hate my guts have gotten to have their overrerpresented voices drive the public image of my
teaching, and there is not a damned thing I can do about it. And, no, I am not going to tame down my wild man/
very tough grading approach that gets me those reactions from a small, but very vocal, minority of students.
BTW, another way to look at these is to do a ratio of the overall number to the "toughness" number, keeping in
mind that "1" is actually zero. That is a way of correcting for how toughness tends to drag down the overall number.
Posted by: Barkley Rosser at Apr 25, 2008 5:33:07 PM
Indiana Jim,
Are you a good teacher or not? This would help assess whether the comments are predictive or not.
Posted by: D at Apr 25, 2008 5:37:33 PM
I'm a seasoned student--several undergraduate institutions and graduate school--and in my experience the single best indicator of professor quality is simply whether or not they make their own lecture notes. The ones who do are nearly invariably above average, and the more detailed the notes the better the professor, the more sincerely and explicitly they wish to convey their body of knowledge.
Most students, though, evaluate professors based on about a 50/50 weight of whether or not they: 1) Enjoyed the class and 2) Got good grades. To me, this argues powerfully for the signaling model of education because post-enrollment, these are the two key variables that determine your utility gain from the class (1 determines cost and 2 determines the change in the strength of the ability signal). Neither has much to do with much you learned.
I should say--as someone whose gotten pretty good grades even in advanced, graduate level math classes--that most professors who are notably harder than other profs in their institution/department are just arrogant jerks. They think that there is some real level of difficulty that students ought to experience, but without standardized testing this is just sheer nonsense. You need a large comparison population in order to be able to say that a test really wasn't as difficult and that some students deserve low grades. All profs grade on a curve, the jerks just enjoy making the curves so that most students suffer.
As well, the profs who don't make lecture notes are also much more likely to be the hard-grading jerks. This is both because they are not nearly as good at their job as they think and because the lecture notes make information more available for students.
Most profs, though, don't make lecture notes or prepare for class much and delude themselves that it is somehow their students' fault when they're not understood.
Posted by: kadkool at Apr 25, 2008 6:41:39 PM
I'm a seasoned student--several undergraduate institutions and graduate school--and in my experience the single best indicator of professor quality is simply whether or not they make their own lecture notes. The ones who do are nearly invariably above average, and the more detailed the notes the better the professor, the more sincerely and explicitly they wish to convey their body of knowledge.
Most students, though, evaluate professors based on about a 50/50 weight of whether or not they: 1) Enjoyed the class and 2) Got good grades. To me, this argues powerfully for the signaling model of education because post-enrollment, these are the two key variables that determine your utility gain from the class (1 determines cost and 2 determines the change in the strength of the ability signal). Neither has much to do with much you learned.
I should say--as someone whose gotten pretty good grades even in advanced, graduate level math classes--that most professors who are notably harder than other profs in their institution/department are just arrogant jerks. They think that there is some real level of difficulty that students ought to experience, but without standardized testing this is just sheer nonsense. You need a large comparison population in order to be able to say that a test really wasn't as difficult and that some students deserve low grades. All profs grade on a curve, the jerks just enjoy making the curves so that most students suffer.
As well, the profs who don't make lecture notes are also much more likely to be the hard-grading jerks. This is both because they are not nearly as good at their job as they think and because the lecture notes make information more available for students.
Most profs, though, don't make lecture notes or prepare for class much and delude themselves that it is somehow their students' fault when they're not understood.
Posted by: kadkool at Apr 25, 2008 6:43:09 PM
