What's your P?

Readers of this blog will be familiar with Tyler's insistent question, what's your P?  As in what probability do you put on this event or belief?  (Examples here, here and my favorite here).

Unfortunately this type of thinking is all Greek (I am tempted to say all Arabic, but that would be cruel) to our intelligence services.  Writing in the Washington Post Michael Schrage argues:

It's time to require national security analysts to assign numerical probabilities to their professional estimates and assessments as both a matter of rigor and of record. Policymakers can't weigh the risks associated with their decisions if they can't see how confident analysts are in the evidence and conclusions used to justify those decisions....

[T]he CIA, Defense Intelligence Agency, FBI and the federal government's other analytic agencies have shied away from simple mathematical tools that would let them better weigh conflicting evidence and data. That bureaucratic shortsightedness undermines our ability to even see the dots, let alone connect them.

Consider the National Intelligence Estimates, the Presidential Daily Briefings or many of the critical classified and unclassified analyses flowing through Washington's national security establishment. Key estimates and analytic insights rarely come with explicit probabilities attached. The nation's most knowledgeable experts on the Middle East, counterterrorism, nuclear proliferation, etc., are seldom asked to quantify, in writing, precisely how much confidence they have in their evidence or their conclusions.

Your personal financial planner does a better job, on average, of quantitative risk assessment for your investments than the typical intelligence analyst does for our national security.

All true.  I would add only that one virtue of information markets, like the short-lived Policy Analysis Market, is precisely that they produce such probabilities as a matter of course.