Tim Oren's Due Diligence

Low Rent ETL and Analytics

ETL (extract-transform-load of databases) and formal analytics have long been seen as a competitive weapon employed by large companies with the resources to both collect meaningful data, and manage the process of crunching it. The software segment supplying this market is, if anything, over-supplied. The analytics segment in particular has been in a process of slow motion consolidation for several years. See, for instance, acquisitions by SAS, Informatica, and IBM. Attacking this market with new ventures looks unappealing, given large capital requirements and questionable ROI.

(BTW, I distinguish the general purpose analytics market from web analytics, where you have IBM divesting instead of acquiring, and an very capable free offering from Google that sets a floor under the market.)

Like many parts of the software industry, ETL and analytics have been migrating down market, making tools available to smaller enterprises that have the talent to wield them. The offerings include network based (SaaS) services, and suites of open sourced tools that have been assembled from grass roots projects. Notwithstanding some adoption issues is SaaS, both approaches have the potential to produce offerings better fit to the SMB segment than conventional vendors. So, what's out there? Here's a somewhat raw dump from a recent survey I did in the area.

Further on REST vs. WS-*

RESTful Web Services is a new book out this month from O'Reilly, authored by Leonard Richardson and Sam Ruby. I've got it on order, and will review after reading, but in the meantime here's a nice writeup from Jon Udell, who is a good analyst in this space.

While it seems like much of the book may go further into architectural detail than useful for the non-coder, it also sounds like the framing of the discussion may be well worth the price of admission. In particular, Jon discusses the author's distinction between Resource Oriented Architecture and Service Oriented Architecture, a dichotomy that fits well with my own ubiquity vs. integrity divide on business goals. More after I've absorbed it.

HT: Rich Miller

What's Going On Here?

So what's the deal, you might ask? Nothing but sporadic posts here for months on end, and then suddenly back to something like a regular writing schedule. Here's at least part of the story:

Over a year and a half ago I wrote that I had taken on a part time CEO role at a startup. That venture was in the area of database security, particularly centered on privacy and reidentification control of shared, but matchable databases. We were in stealth mode, and in spite of bold and hopeful promises to the contrary, working on it largely absorbed the creative energy that formerly produced bloggable content, while my actual output was being directed to an internal wiki. (Thanks, Ross!)

We put together some very modest angel funding to allow creation of a working prototype and recruited a capable security-oriented coder to do so. As that approached some definition of ready, I recruited a more business development oriented CEO to take my place, and shop the idea to both early adopters in the finanicial institutions sector, as well potential OEM partners in the security and analytics domains. I continued as CTO during that process.

Long story short: The returns are in, and what we have produced comes out as a feature, in multiple markets, rather than a free standing company. In spite of having written on the topic, sometimes you have to walk the road to find out the answer. Fortunately, we minimized the capital consumed as we did so, but at the end of the day the ability to extract revenue as a stand-alone doesn't measure up to the costs of sales given both the technology and the market. So rather than taking any more investors' money, we have stood down the company and are in the process of looking for a buyer for the technology.

So for the first time in a long time, I've got some free time and head space for blogging and other activities. There's a fair amount of 'scrap research' from the recent venture that will likely find its way here, suitably laundered as always. Pacifica Fund itself is nearing seven years old, and a number of our ventures are going through the exit process. Once these deals have closed, I may be able to write more about the trials and tribulations of the liquidity process without compromising my fiduciary role. Finally, there's a fair number of topics of interest that were laid aside over the last months due to lack of bandwidth, and some of those are already getting back into my reading and writing mix.

Often the best way to find out where you need to go next is to find out what's interesting enough to inspire the energy for researching and writing. So, no long term promises, but I will likely be more active here for at least the immediate future.

Bomb(er)s over Silicon Valley

Locals in the Valley and along the San Francisco Peninsula having been hearing the unfamiliar sounds of heavy, multi-engine piston aircraft from the skies the last few days. A traveling exhibit of WWII American warbirds has been visiting Moffett Field, including a B-25J Mitchell, a B-17 Flying Fortress, and the world's last operational B-24 Liberator. We took a couple of hours to stop by and admire the hardware and restoration work on Sunday. I always find it a surprise to be reminded just how small and cramped the warbirds that defended us back then feel in comparison to modern hardware. At 6'3", I'd have been a poor - or at least extremely uncomfortable - WWII airman. Walking past the preserved P-3 Orion on the way back to the parking lot, and comparing its size and range to the 'Bs' gives testimony to just how fast aviation changed from the 40s to the 50s, when that airframe was designed.

REST for the Rest of Us? The Great Web Services Divide

The investing world's hype around Web 2.0, specifically the Web Services element, often glosses over a deep divide between two approaches to building and providing such services. I alluded to this dichotomy below and thought it might be useful to surface the salient technical differences, as well as the differing business rationales that will likely make the separation a sustainable one. (Deep techies are hereby warned that I'm going to be simplifying madly along the way - bear with me.)

Both of the competing architectures, REST and SOAP/WS-*, are ways of building distributed systems in an open network and software environment. So what's new, isn't that just the Web? Partly, but along the way from vanilla Web to distributed system we move from almost all of the requests flowing from a human's actions with a browser directly into a Web server, to a lot of them being machine-to-machine requests dispatched automatically on behalf of the user, either from server to server, or from programs embedded in a served up Web page, that the user is not likely to realize he is invoking. The notion is to take advantage of the software infrastructure built for the human-to-machine Web to create a experience that is the result of orchestrating computation and communications among multiple devices, blending the results together transparently. Rather than jumping around pages and sites collecting information, the user can have it all pulled together in one place, and that might just be good for business.

DHS on Web Services Security

Having just dished one DHS program, in all fairness I should call attention to a very nice article on Web Services security, hosted by that same agency. It's in a best practices section that's well worth a look. Note that while the article has Web Services in the title, it's talking about SOAP/WS-* style, not REST. This seems to be a commissioned piece; one of the authors also has an interesting blog on security in distributed systems.

Homeland Security: A Tale of Two Systems

Not the usual fodder here, but bear with me. My wife and I decided to sign up for training from the local CERT (Community Emergency Response Team) organization. Not only do we believe in the 'pack, not a herd' philosophy, but we live about three miles from the Big Fault, on the edge of a fire danger red zone, and in one of the wealthiest Bay Area counties that has a grand total of five ambulances available for the whole populace. You work it out - just like in New Orleans, if it's time for the Big One: They aren't coming. So we spent twenty hours learning fun things like triage, structural damage assessment and light search and rescue. Well worth it, and I recommend the program. But that's not the main point of the story.

As a followup, we were requested to run through some online courses on the Incident Command System and the National Incident Management System. Even volunteers would likely find themselves reporting to or embedded in such a system in the event of catastrophe. So I went through the material, but I can't turn off the management and analysis neurons, so the experience led to some reflections on the philosophies, provenance and likely effectiveness of the two 'systems'.

The Dinochrome Platoon

Via the Blogfather, a well-written piece in the WaPo tells the story of our battlebots and the troops who work with them. It sounds rather familiar to anyone who grew up on Keith Laumer's SF stories of the Bolos, gigantic autonomous battle tanks of the future. Laumer wrote in the 60s, when the images of WWII armor battles still lingered, when counterinsurgency and asymmetric warfare were still minor military specialties, and before the miniaturizing effects of Moore's Law had really kicked in. A real 21st century PackBot would be dust under the ten foot wide treads of a Bolo Mark XXVIII.

What's in common between fiction and fact is the tendency of the warriors to give names, ranks and even decorations to the bots, and go well beyond the call of duty in attempting to preserve and repair them. Perhaps unsurprising given the tendency of teamwork under stress to create strong relationships- and a phenomenon fairly well know in the literature. Back in the day... (flashback dissolve) I and a team of folks at Apple showed that you could get people to attribute human emotions to a 32x32 pixel black and white icon fronting for a simple database algorithm (skip to 'Guides' heading here). Then a couple of Stanford profs systematically showed just how simple it is to get someone to project human motivations and social roles onto computing or communications devices, with very modest amounts of cueing. This is so easy that it's likely an evolved pattern - when something shows even a small amount of the behaviors that we associate with humanity, it gets attributed with the whole boatload, because that's been the safe bet over time.

Death Spiral of the Dead Tree Media, Scene 127

Mike at Techdirt rounds up recent whinging from buggy whip makers newspaper executives complaining about their dying business models. Bossy still has some milk, perhaps they can retire before the cash cow's teats finally dry up.

Meanwhile, an overview of four decades of history of a regional newspaper. Ok, it's satire, so file it under 'fake but accurate'.

As if to underscore both posts, the Minneapolis Star-Tribune scuppers Lileks' column and busts him back to the local news desk, which will be a better use of his talents. I mean, it's not like he's a certifiable genius of a writer or anything.

The bite to all of this is that creative destruction makes no guarantee that creation of new business models will precede apace with the destruction of the old, but breakage of old institutions and ways of earning a living is pretty much guaranteed. It would be nice if we could point to an obvious way for someone of Lilek's caliber to jump onto an alternative source of cash flow based on a survivable, new media model. But, so far, we can't. Each reporter and writer trying to change sides of the media divide is right out there in start up land with the VCs and entrepreneurs.

Update: Jeff Jarvis arrives at the same conclusion, more articulately, but then he's been on GZ for some time.

Update 2: Another great and detailed take, via Rich Skrenta.

Net-based Statistics Course and Calculators

While helping out a friend who had a small analytics problem on hand, I bumped into this nifty set of forms-based statistical calculators hosted at Vassar College. Those also a companion interactive textbook to help you figure out what they are doing, and why, and when you should use them.

This is scaled for student tutorial problems, but as in the case I was working on, there are a lot of useful 'real world' problems that are naturally at that scale and don't warrant the application of high end products like SAS. Consider this kind of thing as an alternative to the 'Excel he**' approach of trying to bend a tool into analytics applications it wasn't designed for. (Also take a look at Dabble for low-rent analytics usage.)