What’s your P?

Readers of this blog will be familiar with Tyler’s insistent question, what’s your P?  As in what probability do you put on this event or belief?  (Examples here, here and my favorite here).

Unfortunately this type of thinking is all Greek (I am tempted to say all Arabic, but that would be cruel) to our intelligence services.  Writing in the Washington Post Michael Schrage argues:

It’s time to require national security analysts to assign numerical
probabilities to their professional estimates and assessments as both a
matter of rigor and of record. Policymakers can’t weigh the risks
associated with their decisions if they can’t see how confident
analysts are in the evidence and conclusions used to justify those
decisions….

[T]he CIA, Defense Intelligence
Agency, FBI and the federal government’s other analytic agencies have
shied away from simple mathematical tools that would let them better
weigh conflicting evidence and data. That bureaucratic shortsightedness
undermines our ability to even see the dots, let alone connect them.

Consider the National Intelligence Estimates, the
Presidential Daily Briefings or many of the critical classified and
unclassified analyses flowing through Washington’s national security
establishment. Key estimates and analytic insights rarely come with
explicit probabilities attached. The nation’s most knowledgeable
experts on the Middle East, counterterrorism, nuclear proliferation,
etc., are seldom asked to quantify, in writing, precisely how much
confidence they have in their evidence or their conclusions.

Your
personal financial planner does a better job, on average, of
quantitative risk assessment for your investments than the typical
intelligence analyst does for our national security.

All true.  I would add only that one virtue of information markets, like the short-lived Policy Analysis Market, is precisely that they produce such probabilities as a matter of course.