« The Education of the Stoic | Main | The substitution of capital for labor, charity edition, seen in Atlanta »
Debating Economics
Intelligence Squared has held a series of debates in which they poll ayes and nayes before and after. How should we expect opinion to change with such debates? Let's assume that the debate teams are evenly matched on average (since any debate resolution can be written in either the affirmative or negative this seems a weak assumption). If so, then we ought to expect a random walk; that is, sometimes the aye team will be stronger and support for their position will grow (aye after - aye before will increase) and sometimes the nay team will be stronger and support for their position will grow. On average, however, we ought to expect that if it's 30% aye and 70% nay going in then it ought to be 30% aye and 70% nay going out, again, on average. Another way of saying this is that new information, by definition, should not swing your view systematically one way or the other.
Alas, the data refute this position. The graph shown below (click to enlarge) looks at the percentage of ayes and nayes among the decided before and after. The hypothesis says the data should lie around the 45 degree line. Yet, there is a clear tendency for the minority position to gain adherents - that is, there is an underdog advantage so positions with less than 50% of the ayes before tend to increase in adherents and positions with greater than 50% ayes tend to lose adherents. What could explain this?
I see two plausible possibilities.
1) If the side with the larger numbers has weaker adherents they could be more likely to change their mind.
2) The undecided are key and the undecided are lying.
For case 1, imagine that 10% of each group changes their minds; since 10% of a larger number is more switchers this could generate the data. The problem with 1 and with the data more generally is that we don't seem to see a tendency towards 50:50 in the world. We focus on disputes, of course, but more often we reach some consensus (the moon is not made of blue cheese, voodoo doesn't work and so forth).
Thus 2 is my best guess. Note first that the number of "undecided" swing massively in these debates and in every case the number of undecided goes down a lot, itself peculiar if people are rational Bayesians. A big swing in undecided votes is quite odd for two additional reasons. First, when Justice Roberts said he'd never really thought about the constitutionality of abortion people were incredulous. Similarly, could 30% of the audience (in a debate in which Tyler recently participated (pdf)) be truly undecided about whether "it is wrong to pay for sex"? Second, and even more doubtful, could it be that 30% of the people at the debate were undecided--thus had not heard arguments in let's say the previous 10 years that converted them one way or the other--but on that very night a majority of the undecided were at last pushed into the decided camp? I think not, thus I think lying best explains the data.
Some questions for readers. Can you think of another hypothesis to explain the data? Can you think of a way of testing competing hypotheses? And does anyone know of a larger database of debate decisions with ayes, nayes and undecided before and after?
Hat tip to Robin for suggesting that there might be a tendency to 50:50, Bryan and Tyler for discussion and Robin for collecting the data.
Posted by Alex Tabarrok on April 24, 2009 at 07:41 AM in Data Source, Economics, Education, Games, Sports | Permalink
Comments
People who are weakly attached to their position, or have no particular grounds for belief one way or the other, end up with the majority for social reasons, and these are the people easiest to sway? (Related to your #1, but from a different direction.) It seems like people who hold an unpopular position are going to either need very good reasons for doing so, enough to cut against social pressure (in which case your debate is less likely to supply arguments able to sway them), or are going to be the kind of cantankerous and contrary people who don't care what others thing (in which case your debate is irrelevant to them anyway).
Posted by: Andromeda at Apr 24, 2009 7:44:07 AM
Seriously unbalanced starting opinions may also be the result of imbalances in the main stream of discourse. If 70% of people favor measure X, it probably means that at least 40% of people have never heard an intelligible argument against X. The debate may be the first exposure people have to the people opposing X who were not straw men or media caricatures.
Posted by: Grant Gould at Apr 24, 2009 7:54:12 AM
Actually, Alex, I think that concluding undecideds are lying reflects a remarkably obtuse and/or cynical view of human nature. There may in fact be a significant portion of undecideds in any context who already have a decided opinion one way or another, but choose to lie for a number of reasons, including perhaps concern over what other people might think of them, especially if it turns out the consensus post-debate is that they are "wrong."
But I can think of several other explanations off the top of my head. One: the undecided really does have a considered opinion, which he will remember when he thinks about it, but it is buried away in his memory and he literally can't remember it when the pollster asks him. This strikes me as plausible when the undecided is asked his opinion on a subject which he does not normally think about. It may be decided in his mind, but he does not think about it regularly, so it takes time to recall his stand. When unprepared to give an answer, he declares himself undecided and later "remembers" how he feels. I imagine there are vast swathes of questions even highly intelligent academics who pride themselves on thinking for a living would respond to in similar fashion.
Two: the undecided has a bias, or prejudice about the question, but he realizes that his opinion, while firm, is not based on adequate consideration or evidence. He decides to declare himself undecided and listen to the arguments pro and con, to make sure that when he does decide, he has done so after thinking about and hearing different arguments in a much more systematic fashion. Far from being dishonest, this potential stance is in fact steeped in epistemic honesty. "I think I know how I feel about this, but honestly, I know I haven't formed my opinion through careful, systematic evaluation of the pros and cons. Educate me." Your example of how people feel about paying for sex seems a perfect instance of just such a question: after all, everyone already "knows" it is bad to pay for sex, right? Of course, there will be various levels of conscious or unconscious honesty in these instances. Perhaps the undecided pretends to himself he will listen fairly, but simply listens for arguments to confirm his biases rather than taking on board the full range of arguments in a balanced manner.
Three: there are in fact intelligent people who think deeply about issues who prefer to maintain a stance of epistemic agnosticism about certain issues. Justice Roberts and abortion may in fact be a good example of this (I do not know). Such people reserve judgment on important issues until they can take the time to listen to competing viewpoints, weigh the arguments, and come to a reasoned conclusion. All this takes time, thought, and energy, which most people cannot easily spare from their busy lives. There are vast swathes of issues in my intellectual lumber yard, for instance, that I have filed under "Have not decided; review when opportunity presents itself." This is not lying. This is radical epistemic honesty.
Four: there are people who have considered opinions on a subject but who are open to listening to competing arguments and perhaps even new facts. J.M. Keynes comes to mind. They have thought and decided, but they are open to being persuaded otherwise. Of course, they do not have to declare themselves undecided: even a decided listener can change his mind after a debate. But perhaps there is an element of intellectual challenge and incentive-setting they give the debaters by declaring themselves undecided. "Go ahead, big guy, change my mind." While perhaps less honest than arrogant, such a stance is at its base fundamentally an honest one.
Lastly, anyone who thinks they know all the answers all the time, who thinks decisions they made on issues twenty years ago in graduate school are necessarily correct and require no further thought or consideration, are fools and intellectual charlatans. That is the truly dishonest position in this entire debate.
Posted by: The Epicurean Dealmaker at Apr 24, 2009 8:06:05 AM
It may also have to do with audience bias towards debating rules. While the preliminary vote is purely based on the issues, the final vote will be basedon that plus a variable based on perceptions on how the arguments where presented. Since it is unusual for one team to completely wash, a shift towards the underdog makes sense as does that this effect is incomplete, as the partisan maintain their position. Note that this may only apply to debates with no real stakes.
In brief, switching potentially signals one as having a level of debate experience or a level of uncertainty. Can these effects be separated from the aggregate data level.
Posted by: Leo at Apr 24, 2009 8:08:09 AM
Your assumption "the debate teams are evenly matched on average since any debate resolution can be written in either the affirmative or negative" seems wrong for this series: There appears to be a tendency to formulate in favour of "ayes", especially for extremer cases.
There are a lot of "after-ayes" over 60%, but only one below 40%
This is somewhat true if your graph shows "ayes" as percentage of total decided votes. If your graph shows "ayes" as percentage of all votes, including undecided, then the effect is even stronger.
Posted by: Zamfir at Apr 24, 2009 8:15:27 AM
I see that it is as percent of decided votes, my apologies for missing that part of the text. Still, winning votes show much stronger margins than losing votes. Perhaps we can ask the organizers whether they flipped a coin to decide who got the aye-formulation?
Posted by: Zamfir at Apr 24, 2009 8:22:21 AM
Do the debaters see the pre-debate votes? If so, it could be that those debaters who are on the underdog side of the issue try harder to convince the audience, since they know they have more people to win over.
Posted by: Ben at Apr 24, 2009 8:27:31 AM
I suspect a lot of people hold the majority opinion simply because it is the majority opinion. This is not necessarily irrational: they haven't had time to look at arguments carefully, they figure that if most people believe P, then probably P, etc. The debate context has the effect of focusing attention on arguments and weakening all the reasons to hold an opinion just because it is the majority's. So the majority side, on net, will lose voters in such cases.
Posted by: Heath White at Apr 24, 2009 8:27:51 AM
Maybe most people are just credulous, weak-minded, and easily swayed? Andromeda makes a good point: maybe weakly-attached people are most easily persuaded one way or another. There's also the question of whether people will change their minds out of sympathy for the underdog team.
Posted by: Neal at Apr 24, 2009 8:39:23 AM
Grant makes a good point above. Is it usual that the minority position is less known? If it is less known, perhaps the "newness" of the argument can sway them. Perhaps this could be monitored and controlled for by testing the knowledge of those polled both before and after.
Posted by: Jacob at Apr 24, 2009 8:50:26 AM
A commenter states:
"Actually, Alex, I think that concluding undecideds are lying reflects a remarkably obtuse and/or cynical view of human nature. There may in fact be a significant portion of undecideds in any context who already have a decided opinion one way or another, but choose to lie for a number of reasons, including perhaps concern over what other people might think of them, especially if it turns out the consensus post-debate is that they are "wrong."
I am not sure that Alex meant to imply ill intent when he used the word "lying" (I could be wrong). I read it as just a factual statement that the best reason for the discrepancy is that folks claim to be undecided when in fact they really lean one way or another albeit maybe not very strongly. They might have a "good" reason for "lying" but from an objective point of view the breakdown in the study comes from folks not properly expressing their starting views.
I agree with both of you. I agree that "lying" is probably the cause of the discrepancy but I also think that the reason folks are "lying" is not malicious.
The best cure for this, in my opinion, is to better phrase the starting question by doing one of the following:
(1) Offer 5 options: Strongly Agree, Weakly Agree, Undecided, Weakly Disagree, Strongly Disagree
(2) Qualify the question with a statement that says "Only answer Undecided if you are completely unsure of your stance. If you lean slightly towards Agreeing or Disagreeing, even if you feel that your view is not well enough informed, you should pick a side rather than stating Undecided."
Posted by: mooshinator at Apr 24, 2009 8:57:26 AM
Yeah Alex, I too think you jumped way too quickly to, "These people are lying." I wasn't surprised by the results you show, and my vague explanation (which is similar to the first two posters') is that a lot of people probably thought something was so "obvious" that they never really pursued it much. And then that very night, they heard some arguments that made them say, "Huh, I never thought of that!"
It would be interesting to wait a month and then ask the same people. Maybe they were startled by how decent a case the other side put up, but after time they reconsidered and went back to their old opinion.
Posted by: Bob Murphy at Apr 24, 2009 9:00:42 AM
since Y is just a lagged version of X i'm thinking regression to the mean. if you assume that the debate organizers try to avoid topics like "child-molesting, pro or con" and choose well-balanced subjects then the true prior should be about 50% and observed pre-debate deviations from this are mostly picking up random error. the next time you measure opinion on the same issue the random error comes out differently and this appears to be a "pro-minority effect" when really it's just an artifact of the study design.
since most people have low salience and low information on most issues and panel surveys show low reliability in attitude measurement i think it is reasonable to assume there is a large random component to opinions.
Posted by: Gabriel Rossman at Apr 24, 2009 9:07:07 AM
Do note that being undecided in some sort of vague, haven't really thought about it way is not enough. The undecided swing to the underdog. If it were truly "hadn't really thought about it" we would surely expect the undecided to mirror the decided, ala the random walk hypothesis.
I thank mooshinator for his generous comments.
Alex
Posted by: Alex Tabarrok at Apr 24, 2009 9:16:33 AM
When reading it, my reaction was similar to that of Bob Murphy above.
The very nature of indecision is that you alternate between being decided one way and being decided the other way. In the immediate aftermath of a well presented argument it's natural to lean that way.
Posted by: darynr at Apr 24, 2009 9:29:57 AM
If one goes into a debate with the assumption that the participants are of roughly equal skill levels, it's easy to attribute a "win" to the idea that the winner had more to work with i.e. the winner was arguing the right position.
In the real world, arguments for and against a particular position would be encountered continually (probably with the same average level of argument quality), effectively canceling each other out as far as the undecided are concerned.
In a debate, on the other hand, we are encouraged to regard each debater as the definitive expert on his/her side of the issue.
Posted by: darynr at Apr 24, 2009 9:47:31 AM
Looks like a simple variation on "regression to the mean" to me.
Each side has equally talented "convincers", and so will be equally successful at flipping the other side's weaker, marginal adherents.
That will always help the side with the least support to begin with, because it has fewer marginals to flip, while the side with the most support has more.
Thus, if the initial distribution is 70/30, with equally talented debaters I'd always expect the 70-side to lose support and the 30-side to gain it. Only with an initial 50/50 distribution would I expect no change.
Posted by: Jim Glass at Apr 24, 2009 9:49:21 AM
Maybe people attend these things because they want to be persuaded by an unusual view point.
Posted by: josh at Apr 24, 2009 9:52:51 AM
Here's a data point: I have a considered opinion on whether it's wrong to pay for sex, but I'm not sure what my answer would be - pigeon-holing a nuanced view into a simple statement is never easy.
I do not think it should be illegal to pay for sex.
I would not pay cash for sex, unless maybe it were to a close friend who was turned on by being treated like a prostitute.
I would not think ill of somebody for offering/accepting money for sex (or the ill thinking would be barely worth a thought).
The thought of sex involving a monetary exchange feels somewhat unpleasant to me.
So is it "wrong"?
Posted by: Jens Fiederer at Apr 24, 2009 9:54:10 AM
I think there is a simplier explanation:
The situation could be explained by an equal percentage of people from each group changing their mind. If you start with 100 people (70 aye and 30 nay) and an equal percentage of each group changes positions, for example 20%, then after the debate you have 61 aye and 39 nay.
The minority gains adherents because of the imbalance at the beginning of the debate.
Posted by: nsp at Apr 24, 2009 9:56:20 AM
I meant to add, relative to the "regression to the mean", explanation, that I don't see how this is an objection to it...
The problem with 1 and with the data more generally is that we don't seem to see a tendency towards 50:50 in the world. We focus on disputes, of course, but more often we reach some consensus (the moon is not made of blue cheese, voodoo doesn't work and so forth).
... as we aren't talking about "the world" and questions that in time can be factually answered definitively ("is the moon made of cheese?") -- we are talking simply of these debates.
And actually there is a lot of tendency towards 50:50 in the real world, if you look where you are likely to find it. For instance, the two major political parties tend to split 50:50 over time, as the result of endless argument and debate.
Posted by: Jim Glass at Apr 24, 2009 9:59:09 AM
This is how I've always explained this to myself:
I think that one can hold a position for two different reasons: either they've thought through the arguments as best as they could and come to a rational conclusion, OR they've simply jumped to the conclusion that intuitively seemed correct to them. (I do not mean that they jumped to it in the moment when presented with the initial voting -- it could have happened years ago.)
The people in that latter group will tend to be particularly likely to change their mind upon having the arguments on both sides laid out for them.
SO, if we assume that 50% of people fall into this category for any given issue (your willingness to go along with that figure will depend on your faith in folks' antiintellectualism, I suppose), and that 50% of those folks will change their mind after hearing the arguments, then you would expect that a 30% / 70% beginning split would end up at 40% / 60% -- one quarter of each group switches to the other group!
Posted by: alesh at Apr 24, 2009 10:01:22 AM
As Gabriel Rossman points out, regression to the mean is almost surely part of the story.
Suppose that the "before" position is 90% "For". Then for every 1 person the "against" persuades, the "for" would have to persuade 9, just to break even. If undecideds break 50-50, then the after will be closer to 50-50 than the before. If equal shares of the "decideds" also switch then that also leads to regression to the mean. The "for" has a much tougher job if maintaining the same share is the standard.
The extreme version is perhaps even more obvious: suppose that "before" was 100% to 0%. Then it is mathematically impossible for the "after" to be more extreme but it could be more evenly split.
Posted by: a student of economics at Apr 24, 2009 10:12:39 AM
As Gabriel Rossman points out, regression to the mean is almost surely part of the story.
Suppose that the "before" position is 90% "For". Then for every 1 person the "against" persuades, the "for" would have to persuade 9, just to break even. If undecideds break 50-50, then the after will be closer to 50-50 than the before. If equal shares of the "decideds" also switch then that also leads to regression to the mean. The "for" has a much tougher job if maintaining the same share is the standard.
The extreme version is perhaps even more obvious: suppose that "before" was 100% to 0%. Then it is mathematically impossible for the "after" to be more extreme but it could be more evenly split.
Posted by: a student of economics at Apr 24, 2009 10:13:17 AM
I would guess that the before-majority tends to reflect a consensus opinion, the arguments in favor of which are probably more commonly encountered in everyday life. Most of the arguments and ideas that are new to the audience would then be coming from the anti-consensus side, so more weakly-decided people would break for the side bringing previously unconsidered information to the table.
Posted by: MikeF at Apr 24, 2009 10:13:19 AM
