« Bill Traylor | Main | What Credit Crunch? »
Does the quality of blog comments deteriorate?
Forget about MR and its superb commentators, I am talking about the typical above-average blogs. I often have the impression that the best comments come in the first fifteen or so, after which quality declines precipitously and often exponentially. Why might that be?
1. The truly smart people only like to make smart points on "fresh" posts. For instance more people read the comments on fresh posts (but why?), so the benefit of a quality comment is lower as the post becomes older.
2. As time passes, the chance that a warring twosome find each other, and take over the thread, increases.
3. There is a tendency to attack or respond to the stupidest or most controversial thing said, and the longer the comments thread runs for, the stupider this will get.
4. As the number of comments multiplies, so does the number of independent discussion threads and the optimal number of threads is exceeded.
5. (Addended) As one (early) commentator notes below, the simple fact of diminishing marginal utility.
Might some of these mechanisms also help explain why a) history of thought is "ghettoized" as a field, and b) there is such a high premium to working in hot, new fields? The general point is that there are increasing returns to scale for high quality discussions; furthermore those quality discussions are quite fragile and require cultivation and subsidization through norms. Freshness matters, so stale topics will indeed encounter discrimination.
Comments are open, who wants to go first?
Posted by Tyler Cowen on February 5, 2008 at 06:49 AM in Web/Tech | Permalink
Comments
It's an election year.
Posted by: Edgardo at Feb 5, 2008 7:59:50 AM
Some posts do not need a lot of discussion. Rational bloggers will comment as long as the benefit exceeds the cost of further discussion. Rational bloggers will exit the thread leaving those with a low opportunity cost of time.
Posted by: Mike Fladlien at Feb 5, 2008 8:00:59 AM
Geez, I don't know about this one. What about something like Making Light?
There, open thread can have 500+ comments, and the thread can elicit interesting conversation beginning at comments in the 300s. And topical posts (many fewer per day than here) can have long and substantive discussion throughout.
So I'm putting in for two more explanatory variables: moderator/community norms and a more gentle post-pacing.
Posted by: Peter at Feb 5, 2008 8:11:53 AM
There's also an inverse to point #1-- trolls often turn up at the ends of long comment threads posts because they don't bother to read earlier posts.
Posted by: MattF at Feb 5, 2008 8:14:11 AM
I think trolls often turn up at the ends of long comment threads posts because they don't bother to read earlier posts.
Just kidding. I think the progression is: point made, conversation, argument, flame war.
Maybe the "points" are raised early.
Posted by: odograph at Feb 5, 2008 8:27:18 AM
Is there any discernible difference in the quality of posts between folks who use pseudonyms v first names v full names?
Posted by: burger flipper at Feb 5, 2008 8:34:11 AM
Fresh posts have fewer comments to read. Most commenters feel that they must read the entire comment thread before commenting (why else, after all, is the "Post a comment" form at the bottom of the page?). This means that late comments will be of two sorts: Those that are of sufficiently high value to the poster to be worth reading all of the others, and those by people who have already read most of the comment thread in order to post previously. Both of these alternatives favor narcissists and flamers over people with other priorities.
Hypothesis: The quality of comments (but not of the discussion as a whole?) could be improved by moving the "Post a comment" form to above the comments. This would discourage give-and-take among commenters, though, so such a system would probably be best accompanied by some sort of administrative promotion of important comments -- a "comments of the week" thread like some blogs have taken to doing, perhaps.
Hypothesis: The quality of comments could be improved by capping the number of comments. Perhaps after the first fifty comments only people who have not yet posted on the thread should be allowed, and after a hundred no further comments should be allowed at all.
Posted by: Grant Gould at Feb 5, 2008 8:35:04 AM
Unfogged keeps it going well past the point where other blogs have stale comments. But maybe they're not playing the same game as you.
Posted by: Justin at Feb 5, 2008 8:36:28 AM
Interesting to think that the ghettoization of thought is caused by the
same mechanisms that apply to the unravelling of a coherent thread. Though
it seems to me that this gehttoization is due more to the fact that people
have to think in discrete categories. Our capacity for consilience is
pretty limited.
Points two and three, driven by ego, probably account for some convoluted
and contradictory theories in economics and the soft sciences. Maybe
it's not so applicable to the hard sciences. But look at Aristotle.
Also, when everything has already been said and done, you gotta make stuff
up to look smart. Look at anthropologists.
Posted by: Probiscus at Feb 5, 2008 8:39:17 AM
like...duh...whatever.
Posted by: Caravaggio at Feb 5, 2008 8:47:38 AM
I agree. This comment is an example.
Posted by: Zettel at Feb 5, 2008 8:49:27 AM
The precipitous decline has already started....ah, the joys of reflexivity!
Posted by: Caravaggio at Feb 5, 2008 8:49:40 AM
I'm so pissed off that there are already 9 comments here that I can't even remember the important point I had to make. What do you people do? Hover over this site all day do you can be the first to comment one each post?
Posted by: Tom at Feb 5, 2008 8:50:16 AM
I think I am an intelligent person. I tend not to comment on long threads after a post because I do not want to read through to make sure that my point has not already been made, or worse, been made more eloquently by someone else. The effort required in making a point is much much less when you do not need to sift through mounds of potential gibberish.
I suppose you've proven that I should always read at least the first dozen comments though.
Posted by: Jonathan at Feb 5, 2008 8:54:01 AM
It's not that a topic grows stale; it's that it moves elsewhere. For instance, you wrote an entry about the quality of Tide recently. I was away for some time so I didn't read about it until days after it was posted, enough time for the comments to add up. The best comment--a rough summary report by a marketing analyst--came after the initial flood of comments. But no one responded. Conversation is king. And no one wants to have one in the comments section of a post five days old.
Posted by: Rue Des Quatre Vents at Feb 5, 2008 9:03:00 AM
As with the other Peter, I agree that Making Light tends to have very interesting discussions (where else can one find blends of Lovecraft and Milne). However, Calculated Risk is a blog with multiple posts per day (some of which are very short, and some are rather long) and those discussions tend to get replys in the 100-200 range.
Calculated Risk: http://calculatedrisk.blogspot.com/
>so the benefit of a quality comment is lower as the post becomes older...
At slashdot, the odds of your post getting moderated up to +5 are far greater when you're one of the first posters. Early posting tends to drive the groupthink of the other posting and moderation as well. I suspect that slashdot would be an interesting source of data on group moderation, groupthink and the reward of being the first poster. Some first posters like to post "First!" and one site, the forums at F*d company, used to filter those "first" posts, replace first with "boobies" and re-timestamp the post a couple hours later.
Unsubstantiated theory: At both ML and CR, the original posters reply often during the actual threads, and this steers the conversation, sometimes not very gently.
An example of where all the intelligent discussion happens in about the first 50 posts and the remaining 200-500+ are consumed by trolls: J Kunstler's blog (which has the F word in the title). He posts once per week (early AM monday), and extremely rarely posts responses in his own threads, so the trolls have all week to work.
>...history of thought is "ghettoized" as a field...
Two responses to this comment.
1 - The US has a history of disparaging intellectuals. Pretty much the last time that intellectuals were admired, and maybe not even then, would be in the 1776-1787 timeframe.
2 - I'm going to point you to an essay by Paul Graham that I think is relevant here: http://www.paulgraham.com/marginal.html
And I'd say the relevant portions are around:
>So the first question to ask about a field is how honest its tests are, because this tells you what it means to be an outsider. This tells you how much to trust your instincts when you disagree with authorities, whether it's worth going through the usual channels to become one yourself, and perhaps whether you want to work in this field at all.
>Tests are least hackable when there are consistent standards for quality, and the people running the test really care about its integrity. Admissions to PhD programs in the hard sciences are fairly honest, for example. The professors will get whoever they admit as their own grad students, so they try hard to choose well, and they have a fair amount of data to go on. Whereas undergraduate admissions seem to be much more hackable.
>One way to tell whether a field has consistent standards is the overlap between the leading practitioners and the people who teach the subject in universities. At one end of the scale you have fields like math and physics, where nearly all the teachers are among the best practitioners. In the middle are medicine, law, history, architecture, and computer science, where many are. At the bottom are business, literature, and the visual arts, where there's almost no overlap between the teachers and the leading practitioners. It's this end that gives rise to phrases like "those who can't do, teach."
and
>Where the method of selecting the elite is thoroughly corrupt, most of the good people will be outsiders. In art, for example, the image of the poor, misunderstood genius is not just one possible image of a great artist: it's the standard image. I'm not saying it's correct, incidentally, but it is telling how well this image has stuck. You couldn't make a rap like that stick to math or medicine.
>If it's corrupt enough, a test becomes an anti-test, filtering out the people it should select by making them to do things only the wrong people would do. Popularity in high school seems to be such a test. There are plenty of similar ones in the grownup world. For example, rising up through the hierarchy of the average big company demands an attention to politics few thoughtful people could spare. Someone like Bill Gates can grow a company under him, but it's hard to imagine him having the patience to climb the corporate ladder at General Electric—or Microsoft, actually.
>It's kind of strange when you think about it, because lord-of-the-flies schools and bureaucratic companies are both the default. There are probably a lot of people who go from one to the other and never realize the whole world doesn't work this way.
>I think that's one reason big companies are so often blindsided by startups. People at big companies don't realize the extent to which they live in an environment that is one large, ongoing test for the wrong qualities.
I've worked at too many companies, both large and small, that were completely ignorant of why they were still in business.
Posted by: Peter at Feb 5, 2008 9:12:06 AM
Grant Gould: "The quality of comments could be improved by capping the number of comments. Perhaps after the first fifty comments only people who have not yet posted (...)"
Or maybe make the latest comments (those at the bottom) appear with smaller fonts?
Posted by: puzz at Feb 5, 2008 9:13:01 AM
Maybe not having a linear discussion but a tree might be a solution to the problem that multiple lines of discussions evolve which cannot be finished and fade out. This also might be a solution to the the stupid comments and the warring twosome. Having a discussion tree can leave them sitting and spitting on their own branch forever on.
Posted by: at Feb 5, 2008 9:14:56 AM
I believe you pointed out how the number of comments was highly correlated to the topic. i.e. any post mentioning Krugman elicited more comments that were generally lower in quality.
The tone of the website contributes greatly to the quantity and quality of comments. Tyler's posts are mature, insightful and entertaining. Compare this to CalculatedRisk (great blog). The subject matter and generally negative, satirical tone attracts less constructive feedback and creates an atmosphere where it is more acceptable to spam bandwagon or confrontational posts.
Posted by: Mark at Feb 5, 2008 9:15:07 AM
Better blog design might be a solution. Allow readers to comment on comments by creating sublist as in discussion forums. Let readers rate comments, so we can easily identify interesting comments.
I get increasingly annoyed by the length of comment threads and have almost given up on posting any comments myself. I simply don't have the time to read through all comments ( especially the ones of low quality ) and don't want to repeat previously made arguments. Also the later you post, the less likely you are to be read.
Posted by: authe at Feb 5, 2008 9:25:05 AM
@Jonathon's point above is that there's a tendency for the same comment to be repeated over and again, and a growing likelihood that people won't read the whole thread to see if there are repeats (btw, it would help if comments were numbered so it would be possible to refer back to people, you know, like conversation). The mostly brilliant comic XKCD has an answer - an IRC channel bot that catalogs all the existing logs and chucks out people who make repeat statements. Of course, this is a fancy-but-repeat of my own point that moderation makes a difference, so there is still something left to be desired...
Posted by: Peter L at Feb 5, 2008 9:27:03 AM
As other people have pointed out, ML does a great job of keeping discussions and threads alive and interesting for hundreds of comments. I think there are a several things going on to make that work:
a. The moderation there is very good, supported by a pretty active and interested community.
b. The community there includes a lot of people with widely varying ages, interests, experiences, and beliefs. That can keep new ideas or points coming into the discussion.
The obvious parallel here is with new academic fields; a good way to be an important researcher is to write one of the first few papers in some field. After a field has been investigated by hundreds of very smart, dedicated people for several years, it's a lot harder to make a really interesting new observation or argument. And one exception to this comes when there is a continuous stream of new information coming in--lots of new observations or experiments, engineering/operational experience, people from related fields bringing those fields' tools in, etc.
Posted by: albatross at Feb 5, 2008 9:28:14 AM
If my husband comments at all, it tends to be very early in the thread (as he did above, #7). This is because he is the sort of easily bored information junkie who needs a continual flow of stimulation to his brain.
I hypothesize that such people are more likely to make interesting comments, because their brains are fast-moving and full of exciting, diverse information.
I hypothesize that such people are more likely to use, and frequently check, rss readers, which deliver information to them with maximum speed (because it would be OMG AWFUL if a minute might elapse without new information, if an interesting blog post might be up for a whole minute before you saw it). Therefore they are among the early commenters, because they see your post .3289 seconds after it happens.
Having switched over to rss relatively recently myself, I note that one of the changes in my reading style is I'm much less likely to reread old posts/threads, because they don't show up in my default view. So I imagine the only posts you get late in threads are (a) the sort of deliberative thinker so rare on the internet; (b) people who have gotten obsessed with a particular topic (who are more likely to be raging partisans, chewing on one another like rabid dogs); or (c) spam. Signal:noise not so good here.
Be interesting to see if the time distribution of good comments was different before rss became widespread.
Posted by: Andromeda at Feb 5, 2008 9:30:34 AM
Hitler!
Posted by: Justin at Feb 5, 2008 9:40:25 AM
You better watch out Tyler - you might create a self-fulfilling prophecy by making these comments.
Posted by: Samir Nurmohamed at Feb 5, 2008 9:40:30 AM
