Tim Oren's Due Diligence

Citizen's Media

Jeff Jarvis occupies a fascinating position at the juncture of old and new media. Now, provoked by a speech given by an old friend before the Software & Information Indusry Assocation, he has put up a long post, arguing that blogs are in fact the Genesis Event for citizen's media, which will bypass many of the limits and institutions of existing media. RTWT.

It's an important post, perhaps the more persuasive for its quietness. There is more that can be said in rebuttal to the head of New York Times Digital. Particularly in speaking to the SIIA, the priesthood of professionalized, branded information. For him to doubt that blogs might be the "Pong of Electronic Publishing" is to fail to name the threat. Electronic games may have displaced time and dollars from the traditional media, but citizens' meda are a knife at the jugular of the branded aggregation model that calls itself Publishing (Electronic or not). Here the agenda is clear:

"Ultimately, journalism is about trust. In this new product vision, the aggregator's brand must remain trusted..."

Trust also arises from voice and authenticity, which are characteristics of individuals. Citizens' media allows them to compete on trust and viewpoint, head to head with professionalized mass media. Machine based aggregation lets the reader pick and choose, not the editor.

I get my headlines from Google News and Technorati, I pay attention when Glenn Reynolds does, and I pick up techno-human interest stories from BoingBoing. Every one of them has more influence on me than the editor of the New York Times, whose brand lies sullied in the gutter beside the BBC. Pong? Sword of Damocles, more likely.

A Practical Example of Lack

A Practical Example of Lack of Entry Barrier

From the Social Software Weblog, a likely partial listing of social networking systems:

Affinity Engines,
Amigos.com, href="http://www.asiafriendfinder.com/" target="_blank">AsiaFriendFinder, href="http://www.backwash.com/magazine.php" target="_blank">Backwash, href="http://backwashpets.com/magazine.php" target="_blank">Backwash for Pets, href="http://www.buddybridge.com/" target="_blank">BuddyBridge, target="_blank">BuddyZoo, Buzznet,
Chia Friend, href="http://www.classmates.com/" target="_blank">Classmates.com, href="http://www.communityzero.com/" target="_blank">Community Zero, href="http://www.fastcompany.com/cof/" target="_blank">Company of Friends, href="http://www.theconneck.com/" target="_blank">The Conneck, href="http://www.contactnetworkcorp.com/" target="_blank">Contact Network, href="http://www.corporatealumni.com/" target="_blank">Corporate Alumni, href="http://www.delphiforums.com/" target="_blank">Delphi Forums, title="Dogster" target="_blank">Dogster, target="_blank">Dude Check This Out!, easeek,
ecademy, title="eConozco.com" target="_blank">eConozco.com, target="_blank">eFriendsnet, target="_blank">8minuteDating, Eliyon,
enCentra, target="_blank" title="EntreMate">EntreMate, target="_blank">Eurekster!, target="_blank">everyonesconnected, Evite,
First Tuesday,
FriendFan.com, href="http://friendfinderinc.com/corporate/" target="_blank">FriendFinder, href="http://friendity.de/index.jsp" target="_blank">Friendity, href="http://www.friendsurfer.com/" target="_blank">Friend Surfer, href="http://www.friendsreunited.com/" target="_blank">Friends Reunited, href="http://www.friendster.com/index.jsp" target="_blank">Friendster, href="http://www.friendzy.com/" target="_blank">Friendzy, title="Funchain.com">Funchain.com, target="_blank">GermanFriendFinder, Globe
Alive, GoingProfessional,
gradFinder, href="http://www.growingco.com/" target="_blank">Growth Company, target="_blank">HeiYou, HelloWorld,
hipster, href="http://www.huminity.com/" target="_blank">Huminity, href="http://www.imnotfromhere.com/" target="_blank">I’m Not From Here, href="http://www.indianfriendfinder.com/" target="_blank">IndianFriendFinder, href="http://www.interfacesoftware.com/products/interaction/ia_summary.cfm" target="_blank">InterAction,
ItsNotWhatYouKnow,
KnowMates, href="http://lianqu.com.cn/" target="_blank">LianQu, target="_blank">LinkedIn, Living
Directory, Love.com, href="http://www.thelunchclub.net/" target="_blank">The Lunch Club NYC, href="http://match.com/" target="_blank">Match.com, target="_blank">matcheroo, Mediabistro,
MeetUp, title="MixerMixer" target="_blank">MixerMixer, href="http://network.monster.com/NewEntry.aspx?mode=one" target="_blank">Monster Networking,
mrNeighborhood, href="http://www.myematch.com/" target="_blank">MyEMatch, target="_blank">NetMiner, target="_blank">Netmodular Community, target="_blank">Netparty, target="_blank">Netplaya Burning Man Community, href="http://www.networkingforprofessionals.com/" target="_blank">Networking For Professionals, href="http://www.nerve.com/" target="_blank">Nerve, href="http://www.onlinebusinessnetworking.com/" target="_blank">Online Business Networking Resource,
The Opinion Exchange,
orkut, href="http://www.paljunction.com/" target="_blank">PalJunction, target="_blank">Passion.com, peeps
nation, PowerMingle,
qpengyou, href="http://www.rateordate.com/" target="_blank">RateOrDate, href="http://www.realcontacts.com/" target="_blank">RealContacts, title="RedDate.com" target="_blank">RedDate.com, target="_blank">ReferNet, RepCheck,
Ringo, target="_blank">Ryze, target="_blank">Salesforce.com, target="_blank">SeniorFriendFinder, target="_blank">Shortcut, target="_blank">Silicon Valley Pipeline, target="_blank">Small World Project, target="_blank">Social Circles, Social
Grid, SocialTree, href="http://www.sona.com/" target="_blank">Sona, target="_blank">The Spark, Spoke
Software, StumbleUpon,
target="_blank">Sullivan Executive Networking Community, target="_blank">Talk City, TENG,
There, href="http://www.emode.com/" target="_blank">Tickle by Emode, href="http://cluster.tribe.net/tribe/servlet/" target="_blank">Tribe.net, href="http://www.udate.com/" target="_blank">uDate.com, target="_blank">UUFriends, Visible
Path, Wallop, title="WhizSpark" target="_blank">WhizSpark, target=" _blank=">WisdomBuilder, WiW,
WorldShine,
YeeYoo, href="http://www.yoyonet.cn/" target="_blank">YOYO, target="_blank">Zdarmanet, and target="_blank">Zerodegrees.

(Via Marc Canter]. One could, of course, end up with some big winners anyway, if there are increasing returns on scale of the networks, they are competing for the same users, and there are substantial user switching costs. So far, my own nibbling around the domain, and scanning the comments of a lot of others, doesn't lead me to believe that many of these players have sufficient user value to create such a network effort. (With the possible exception of Friendster.)

Dubious Distinction: Comcast

Dubious Distinction: Comcast

As pretty much anyone in the Bay Area knows, Comcast labels itself as a 'different kind of cable company.' And it needs to try to make that distinction, since it's following in the wake of (and using the same plant as) TCI's legendary low-ball infrastructure build-out, and AT&T's refusal to take responsibility for pretty much anything that happened on their system. Cable has been bleeding customers for years around here - every home on our little cul de sac has a dish.

Considering the nature of Silicon Valley, naturally Comcast has been promoting its cable modem service heavily. Unfortunately, someone forgot to give their ISP marketing and legal crew a clue. Via the Prof, we find that Comcast is cutting off customers for overusing their 'unlimited' service. There's nothing necessarily wrong with defining and charging for different tiers of service - that's a competitive issue - but there's definitely something wrong with telling someone they're buying one thing, and then delivering another, particularly when the change in terms is not even defined. Why, one might even think that instead of actually installing the equipment necessary to implement real tiered service, that they were instead resorting to FUD and misrepresentation!

Comcast has a serious fight on its hand to control customer churn in the limping systems they picked up from AT&T/TCI, and to make some inrounds on those customers lost over the years to satellite. Of course, Comcast's business extends well beyond the Bay Area, but that's the point: all of their current and potential customers can now talk to each other. Pull something bogus in one territory, and the word will quickly spread, undermining the marketing position everywhere.

For bogus implementation and marketing myopia above and beyond the call of duty, Comcast is awarded Due Diligence's mark of Dubious Distinction.

Controlled flight into terrain AOL's

Controlled flight into terrain

AOL's bleeding continues.

The impending heat death of

The impending heat death of Nutch

Metadata is also the reason why Gulker could be wrong, and Nutch in trouble. Counting on the public-minded to make all right has worked with a data product such as Wikipedia, but in that case the OPFOR is largely a bunch of vandals, with little to no other incentives. Open search algorithms sound like a great idea, but there are sadly those who are willing to diddle indexes for a quicky profit at the expense of the commons. Google and its ilk have an ongoing commercial incentive to keep their results - however opaque - useful to an audience, and therefore to defeat the index spammers. Can a pro bono effort hold its own indefinitely against the grind of standing off those with an ongoing monetary incentive to do ill by polluting metadata?

Update: Here's a response that misses the point. It isn't The Man in the form of big businesses that's going to be the problem. They, at least, have a brand and reputation to uphold. (Why do you think I ridicule them? Because it hurts). It's the fly-by-nighters, the no-name resellers and the like, that fill up the first two pages of Google results when I try to find a driver update for my Lexmark laser. It's not the big boys that would do in Nutch, it's the cumulative effects of all those trailer trash.

Metadata arbitrage In a way,

Metadata arbitrage

In a way, that's what Cory is describing.

Shirky on Dean and social software

Shirky on Dean and social software

I may be about the last one on this (real life got the better of me for a few days there), but Clay Shirky has a well reasoned post at Many-to-Many re the failure modes of Dean's use of the Internet in an electoral campaign, with some parallels to my own suspicions. Civilized discussion and lots of bloggy trackback goodness. Bonus Old Fart points for using Apple's ill-fated eWorld to illustrate a point. RTWT.

The State of Machine Translation

The State of Machine Translation

Letting computers substitute for human translators in alleviating mankind's Tower of Babel is one of the most frequently hyped and recycled 'futures'. But, after plugging away for 50 years, what's been accomplished is rather modest. Market analysts predict that in 2007, machine translation (MT) will remain an insignificant one percent of an over $10b translation marketplace. What's gone wrong, and could it change?

The commercial state of MT practice is generally founded on 'rule based systems', a technology that has been in market for over a decade. Modestly successful companies such as Systran of France are built on such technology, which is the basis for most free Internet translators such as Babelfish. However, this generation of MT has a relatively high error rate, particularly when faced with fragmentary or colloquial inputs. This has limited its acceptance to market segments - such as 'free' services - that are willing to accept the modest quality of output that is euphemistically referred to as 'gisting'. The same error rate has also kept MT out of most human translation firms: the burden of proofing and fixing the numerous output errors is similar to that of doing a clean translation. Could this level of acceptance change?

There are two incipient trends that may indicate movement. In the first, large (human) translation bureaus are at least hedging their bets with MT. In the past few years, the business of language translation has consolidated with a few large firms such as SDL and Bowne emerging as the survivors, with service as their primary revenue model. In the last few years, these companies have begun to at least explore the MT portion of the market:

The sale of the Transcend technology to SDL in February was a bellwether of things to come. The Barcelona system was acquired by Bowne Global Solutions, and Lionbridge has partnered with Sail Labs to deploy, develop, and comarket NLP technologies and services. SDL and Bowne Global Solutions appear to have plans for continued marketing of systems to external users, although not necessarily in the retail "shrink-wrap" market.

(This except is from a report generally skewed toward Systran. Check here for another overview from one of the same authors.)

At least one of these acquiring companies, SDL, has continued to sell its acquired MT technology. The degree to which they and other service model translation businesses have attempted to integrate with their human workflow is undisclosed.

A second change is a shift in the research front in MT. A decade ago, much work was going into interlingua approaches to MT, in which any human langauge would first be converted into an abstract representation, then output into any other natural language. As the limits of rule based translation became apparent, and computer power continued to increase, efforts shifted toward 'corpus-based' approaches to MT (and other natural language tasks). In this approach, a very large collection of parallel, aligned texts in the two languages under consideration are analyzed, and used to develop a statistical model which is then applied for translation of previously unseen texts. Many of these approaches are based on hidden Markov algorithms similar to those that have shown some success in speech recognition, and similarly, they require a large stock of training data.

This research front is now sufficiently advanced that DARPA can have a shoot-off between over a dozen academic and commercial research groups. Small startup companies, such as Language Weaver (a USC/ISI spinoff) and Meaningful Machines have emerged in the area. With much of the applied research backed by military and intelligence sponsors, there's not as yet much commercial experience with the new generation of technology. Can it break MT out of market niches satisfied with 'gisting', and get MT into the human translation workflow? The jury is still out. One things for sure: If an organization is already paying for human translation of techdocs, website, or anything else, it should now be keeping a database of these parallel texts. They could be quite useful in a few years.

Does the US military need

Does the US military need social software?

Robert Kaplan and the ChicagoBoyz speculate that one of the side bennies of allied involvement in Iraq is networking with command personnel from other countries. Better get Spoke or something of the sort to keep track of all those foxhole partnerships. (Worth following the link just for the awesome Mongolian Army dress uniforms.)

Hiring in Startups I've been

Hiring in Startups

I've been giving Ross Mayfield some grief of late, so time to balance by pointing out his excellent post on risks and needs of hiring in startups. I'll mention one more thing to beware - what I call 'hiring down': managers who consciously or unconsciously hire those who are not potentially threatening (intellectually or politically). Big companies in fat times can survive some amount of this behavior, a new venture cannot. If you look at a startup where the CEO feels much stronger than all of his reports, you're seeing a failure in the making.