« *Fanfare* meta-list for recommended classical music recordings, 2009 | Main | A public choice theory of the filibuster »
False theories of bargaining power
The idea is that publishers could use their robots.txt as a ransom note, selling it to the highest bidder -- Bing or Google.
Here is the link. Still, it is a cute idea.
Posted by Tyler Cowen on November 11, 2009 at 04:01 PM in Web/Tech | Permalink
Comments
Well put.
Posted by: James Grimmelmann at Nov 11, 2009 5:37:18 PM
There is no reason that a search engine needs to obey robots.txt. It is just a polite request not crawl a particular domain or page. If a publisher really wanted to exclude a search engine and the engine didn't want to be excluded the publisher would need to identify crawling traffic from normal traffic and reject it. This is not as easy as it sounds especially in the adversarial scenario. Who would win in an arms race between publishers and engines? My money is on the ones who specialize in analyzing Internet traffic.
David
Posted by: David Clausen at Nov 11, 2009 5:37:53 PM
A search engine need not obey robots.txt, but a corporation has to, at least in the part of world where they have courts to enforce such things. ToS can be suitably amended.
Posted by: JSIS at Nov 11, 2009 6:10:20 PM
robots.txt is just a useful convention. You can also block spidering based on user agent and IP range. This is the basis of cloaking
Posted by: Derek Scruggs at Nov 11, 2009 6:16:26 PM
David, search engines all obey robots.txt. That is not going to change, no matter how "adversarial" things become. They all also identify themselves when they crawl. That won't change either.
There are plenty of spambots out there doing all kinds of things, but big public companies like Google and Microsoft are not going to secretly crawl and index sites which are explicitly forbidding them to do so.
Posted by: KenF at Nov 11, 2009 6:18:57 PM
@Tyler
Why such a smarmy and uniformed comment?
As if YOU know how this, the battle between the content providers and the search engines, will play out. As Derek Scruggs points out, content providers can wall their garden and take active, technological steps to keep Google out.
Posted by: Patrik at Nov 11, 2009 6:33:03 PM
They VASTLY overprice the value of the NYT content.
Posted by: Doc Merlin at Nov 12, 2009 1:37:51 AM
@Patrick. I don't know what Tyler thinks, but my reaction was unsullied by any knowledge of the ability of robots.txt to effectively compete: Doesn't any publisher benefit from google far more than google benefits from any publisher?
Posted by: John at Nov 12, 2009 1:50:11 AM
Doesn't any publisher benefit from google far more than google benefits from any publisher?
This publisher believes that is the case.
Posted by: anon publisher at Nov 12, 2009 10:12:25 AM
Doesn't any publisher benefit from google far more than google benefits from any publisher?
It doesn't matter if the publisher benefits from Google more than Google benefits from the publisher. It matters if the publisher benefits from Google sufficiently more than the benefit from no Google. Google has a policy of not accepting payment or paying for search indexing and changing this policy would have significant downstream effects (including, but not limited to reputation). If Google determines that losing one site is worth more than hindering the rest of their business, the publisher will be harmed significantly more. If the WSJ opts out of Google, FT and Bloomberg become the big winners.
Posted by: Mo at Nov 12, 2009 10:30:08 AM
A bit off-topic, but maybe somebody could explain how AP works in this context. With many newspapers, most of the national and international news seems to be a barely edited AP feed. There are few substitutes (e.g. Reuters).
How does the presence of this common supplier play into this sort of game? And, how does this depend on the market power (or lack of) that the common supplier has?
Posted by: Zbicyclist at Nov 12, 2009 10:38:36 AM
"As Derek Scruggs points out, content providers can wall their garden and take active, technological steps to keep Google out."
But this is an absolutely terrible business model- putting significant resources into keeping your content away from viewers? This sort of thing only works when you can rely on the courts to grant large settlements and to pay for enforcement (ie under anti trust/copyright laws). For a company on company battle its totally useless.
As to the idea of the 10 largest media outlets going to a Google competitor- its a simple prisoners dilemma. If 9 go over the 10th can stick with Google and ramp up its traffic at their expense. If they make an agreement to do so you can sure as heck expect to see an anti cartilization lawsuit in the courts.
In terms of economics Google does something extremely valuable for both consumers and producers of news, only the large media outlets have spend decades using copyright laws to ensure their profits and market dominance. Give them a decade or two and some of them will come up with a viable business model.
Posted by: tom at Nov 12, 2009 1:28:02 PM
It's an attempt to get to a different market equilibrium. I think News Corp, etc., have an up-hill battle, but it's not a foregone conclusion. This is especially true as more papers go out of business and the market moves closer to oligopoly. Personally, I like the idea of more free content, but people are entitled to try to make a profit.
I would be interested in more on why this is a false theory. Are you assuming that publishers are in a commodity market?
Posted by: Greg at Nov 12, 2009 3:34:08 PM
Commenter is "Mo" is spot-on with:
"It doesn't matter if the publisher benefits from Google more than Google benefits from the publisher. It matters if the publisher benefits from Google sufficiently more than the benefit from no Google. "
as is "Greg" with
It's an attempt to get to a different market equilibrium.
Who knows how it will shake out? But Tyler's off-the-cuff ignorance is appalling.
@tom
But this is an absolutely terrible business model- putting significant resources into keeping your content away from viewers?
It depends on the cost of producing that content. Providing "free" content to viewers is not without cost. Also, define "viewers'. Those who pay or those who don't?
Posted by: Patrik at Nov 12, 2009 10:52:32 PM
Blackmail is great business. Just look what Google is doing http://lifehacker.com/5409467/youtube-will-soon-block-access-from-set+top-devices
Posted by: T-Zombix at Nov 20, 2009 6:20:44 PM
