John Battelle describes Amazon/Alexa's decision to make its crawl and index of the Web open and available for other commercial uses. And aptly speculates whether this will lead to an explosion of new uses that might force Google and Yahoo to do likewise.
At one level the Old Fart in me wants to point and simply giggle about timesharing coming back again. But at what a scale!
One dollar per CPU hour consumed. $1 per gig of storage used. $1 per 50 gigs of data processed. $1 per gig of data uploaded...
Sure, it could add up, but it's a dramatic illustration of Moore's Law to compare that to costs the last time the model was in vogue.
There's a potentially deeper implication in the shadow side of search. Since search became a major director of traffic on the net, there's been a hidden cat-and-mouse game between the indexers, and those who would pollute the resultant metadata for profit. Keyword stuffing, link farming, blog comment spamming... There is little limit to the desire of those of indifferent ethics to corrupt the index for their own benefit, and then complain when the indexers close the holes.
This has the effect of making sure that search at a significant scale can never again be a standalone technology or product. It's now inherently a process, or a service model if you would, because there must be ongoing defense of the metadata against the gamers. It's this that has made me skeptical of Nutch, which has excellent technology, but no service business model to fund the protection of the index when its exploitation becomes profitable.
In this hidden dual space of search, Amazon's move has a potential benefit as well. Google, for instance, is a monoclone. It has "One Algorithm to Rule Them All" (PageRank) and that becomes a singular point of attack and reverse engineering for dark side assailants. If third party utilization of the Alex/Amazon index becomes more diverse and specialized, it will consequently become a harder target for the index attackers, who have a more diffuse target with less value to be derived by any given exploit.