Futures methodology stirs a political flap
So a couple of Dem senators have their knickers in a twist over a DARPA sponsored project to create a futures market related to Middle East economics, politics, and terrorist events. The site is up and ready to roll, but as far as the good senators are concerned, it's 'grotesque' to bet real money on things like the economic health of Jordan.
Since I've served some time in the forecasting and futurism trenches, I'll try and put some hard information into this debate.
The idea of using a marketplace as a forecasting tool for events as well as commodity prices has been around for awhile, at least twenty years. In the futurist domain, it may have evolved from the earlier Delphi method. Briefly summarized, this uses a panel of 'experts', each with some exposure and view on the topic of forecast. Each responds, usually with a numeric score, to a variety of questions. The responses are combined, and average and variance as used both as a forecasting outcome and a tool for discussion. I've used this in settings as simple as a dozen people in a room, with a questionnaire made up on the spot. More elaborate and formal Delphis are often iterated, with multiple passes of discussion and input, often over a period of time where more research is accumulated and relevant external events occur.
One of the goals of Delphi is to dig out hidden consensus among differing viewpoints on the same topic. While a simple numeric score may seem a blunt object for forecasting, a group of scores creates an interesting analysis tool. For instance, a strong bi-modal distribution several related topics is highly suggested of a paradigm rift. Delphi is also useful to dig out 'sneaking suspicions' that are valuable insights, but may never get aired in a formal group presentation.
There are some lesson that have been learned about using Delphi. One obvious one is that the panel had better all have some expertise, however partial, that bears on the topic. Garbage in, garbage out. Second, it's highly desirable that all inputs are anonymous. 'Scoring points' can create big bias, whether the setting is enterprise or political. Third, the participants must be committed to the process. If everyone has an agenda to bring back a 'happy answer', the whole thing will be bogus. We'll see how these lessons carry forward.
In spite of a number of attempts to automate it, good Delphi is highly dependent on a human facilitator to extract the maximum value of analysis and iteration. Decent facilitators of this and other futures methodologies can get a very good consulting rate [grin]. Perhaps the costs of this over a period of time were a motivation for borrowing some formal methods from the commodities markets to apply to iterated forecasting. The notion seems to have evolved in several places at once, but I first heard it from an old acquaintence, Robin Hanson, under the name Idea Futures. [See Update 2 below] It's hardly a mystery in commercial futures practice - under the aegis of IFTF, I briefed a commercial client on forecasting markets back in the mid-90s. Indeed, many people have already encountered the idea in different guise. In the political domain, there's the Iowa electronic markets that try to forecast elections. For a bit more fun, there's the Hollywood Stock Exchange where movie buffs bet on the success of flicks and careers of stars. (RHIT the studio execs keep an eye on it.)
So if this notion is well known, why is DARPA messing with it? After all, the 'R' stands for 'research.' Now we enter into the application domain, intelligence. Application of forecasting markets to this domain is - so far as I know - novel. What follows is speculation on why and where it's going.
One trend in intelligence gathering and analysis is 'open source' - and we're not talking Linux and GPL here. Simply put, as electronic media have proliferated around the world, more information that is useful for security purposes becomes openly available. The problem is digging out the useful bits, assessing their reliability, synthesizing and analyzing the bits together, then using the results to guide clandestine information gathering if needed.
Open source in this sense seems to have started inside the intelligence agencies, but you don't need a government ID card to play the game. For instance, the NOSI site is essentially a blog that compiles military intelligence with a naval skew into one convenient location. One might also consider Rantburg, run by an ex-intel guy, to be an open source site, with a bit of an attitude.
The government initiative in this area is called Open Source Information System, OSIS. While a lot of its sites have gone off the net post-9/11, there's still a lot you can find. For instance, there's the Foreign Broadcast Information Service, which unsurprisingly transcribes overseas radio and television. One would guess these guys are keeping themselves busy with the wretched situation in Liberia. OSIS also has a web crawler. I noticed it drop by here a few times before they made sure it wouldn't reverse-DNS. (You're welcome to any useful crumbs you find, guys.) The point here is that domestic and commercial analysis is also fodder for open source intelligence.
Now let's put these two trends together and stir. One of the problems of open source is that 'synthesize and analyze' portion. Horribly labor intensive. Very hard to automate even partially, as any search and automatic organization maven can tell you. Highly subject to observer bias when done manually. And, by the way, doing it well likely involves dragging in analysts and collateral information that would violate the compartments that are meant to keep secret intelligence secret.
Enter the forecasting market. Here's a way for outside expertise to be used to aggregate their views of relevant open information into a scoring system, a la Delphi. A market method is used to avoid the labor intensive, and compartment violating, facilitation process. An anonymous market eliminates grandstanding, and lets clandestine side analysts slip quietly into the mix to add their biases, without revealing sources or methods. And playing with real money does focus the mind (as I can attest) and keeps the goal state consistent for everyone. You can look at this whole project as a way of recruiting 10,000 open source analyists, on their own nickel, with no clearances needed.
Just one problem. We don't know if it works. (And if the latest wire story be true, we may never.) Big, rare events such as coups and large terrorist attacks pose difficulties for market mechanisms, as seen in real stock markets (Here's a book on the topic that I admire.) So it makes sense that most of the futures contracts proposed have to do with future states of readily measurable economic variables in the region of interest. One drawback of this method is that the necessary upfront codification of such contracts has the effect of embedding assumptions regarding what's important. But, if you can show that a forecasting market yields useful information with that amount of real world feedback, then one can perhaps adventure into rarer and scarier propositions.
[politics] As to the Democratic senators that find this 'grotesque': I find a certain large hole in lower Manhattan to be infinitely grotesque myself, but it's a fact. If contemplating such acts, their perpetrators and backers, and the driver behind them, is too much for you, then it's time to find a new line of work. Your actions suggest to me that pandering and scoring political points are more important to you than using all the expertise available to us to prevent a future occurrence (and it's not like this is violating anyone's privacy.) You seem unfortunately typical of a wide swath of your party that is not safe to be in control of this nation's power in time of war. You and your ilk are rapidly converting this formerly ticket-splitting, libertarian leaning swing voter into a dedicated enemy of the Democratic party. [/politics]
Update: Hoo-boy. Looks like I wasn't the only blogger to spot this item and reach for the keyboard. Here's Josh Chafetz, Stephen Green, Mitch Berg, and of course The Prof himself. The whole debate sure would be more edifying if the politicos and media commentators involved would make the effort of reading what the market was actually supposed to trade:
- Quarterly contracts based on data indices that track economic health, civil stability, military disposition, and U.S. economic & military involvement in Egypt, Iran, Iraq, Israel, Jordan, Saudi Arabia, Syria, and Turkey
- Quarterly contracts that track global economic and conflict indicators
- Specific possible events (e.g., U.S. recognition of Palestine in the first quarter of 2005
But that would be wrong, I guess. Can't get media attention with economic indexes.
Update 2: Well here's a kick. According to Dan Gillmor, my old acquaintence Robin Hanson was in fact the creator of the intelligence futures market! I'd lost track of him in the meantime, but turns out that he's hanging out at George Mason U. and moonlighting at DARPA. You enjoying DC, Robin?