Saturday, February 21, 2004

Improving the CIA’s analysis

The CIA’s intelligence analysis is a key component of national security decision-making, but there are some problems in its production process that have been highlighted by the controversy over Iraq’s prewar possession of weapons of mass destruction (WMD) (“Tenet supports CIA data, search for Iraqi weapons,” Page 1, Feb. 6). The CIA should implement a number of improvements to its analytic process to provide the president and other top national security policy-makers with intelligence analysis that more accurately reflects what analysts know and — more important — what they don’t know.

As a former CIA analyst, Iknow uncertainty is part of the intelligenceproduction process. Intelligence collected rarely provides analysts with a complete picture of what is occurring in a foreign country. When a CIA analyst taps into all the various data streams the U.S. government funnels into its secure communication system, the first sense is of an overwhelming amount of information about all kinds of topics. When precise information is desired, such as the condition of a foreign country’s WMD program, a CIA analyst cobbles together bits and pieces of information to form a picture or story and frequently discovers many gaps in the data. As a result, an intelligence analyst’s judgment rests on a rickety foundation of assumptions, inferences and educated guesses.

Caveats are necessary in finished intelligence as a way to communicate analytic uncertainty. Intelligence agencies would be performing a disservice to policy-makers if their judgments communicated greater certainty than the analysts possessed. In the case of Iraqi weapons of mass destruction, if the CIA’s analytic caveats effectively captured their uncertainty and relayed this uncertainty to the policy level, there was no intelligence failure at the analytic level. Not making every analytic call correctly is just part of the territory in the broader process of governmental learning, but accurately reflecting uncertainties when they exist is crucial.

Unfortunately, caveats also complicate assessments of intelligence accuracy. Words such as “probably,” “likely” and “may” are scattered throughout intelligence publications and prevent easy assessment of accuracy. For example, if CIA analysts had said Iraq probably had weapons of mass destruction, would that analysis be considered accurate or inaccurate? There is no way to tell, given the use of the word “probably,” which communicated the analyst’s uncertainty. Removing caveats for sake of simplicity in assessing intelligence accuracy also unfairly removes the record of analytic uncertainty and, in the end, assesses something with which the analyst never would have agreed. Therefore, all assessments of intelligence reporting must be very careful in their use of accuracy as a measuring stick, and it is for this reason that CIA Director George J. Tenet said, “In the intelligence business, you are almost never completely wrong or completely right.”

So if intelligence does not provide policy-makers with answers, what does it provide for them?

Intelligence agencies provide policy-makers with intelligence reports that help them make decisions in two ways: by informing them of what is going on in foreign countries and by warning them of possible developments that could harm U.S. interests. Unfortunately, the CIA’s performance in both areas suffers from a variety of problems and could be improved.

The CIA could improve its provision of information to policy-makersbycreating processes to ensure that the information provided is accurate. There is a difference between inaccuracies in analytic judgment and inaccuracies in fact. Inaccuracies in judgment cannot be avoided and are limited only through the use of caveats. Inaccuracies in fact are an indication of systemic institutional carelessness. The CIA fails to rigorously fact-check its intelligence products, and the consequences of this were highlighted in the 1999 bombing of the Chinese Embassy in Belgrade, Yugoslavia. In that case, the CIA failed to ensure that the legitimate Yugoslavian target was identified and located before relaying the targeting information to the military, but this kind of inaccuracy is prevalent throughout all levels of intelligence reporting.

For intelligence information, validation requires two levels of fact-checking. The first is to ensure that the finished intelligence accurately captures the information contained in the raw intelligence reports provided by collection agencies; the second level is to ensure that the raw intelligence reports themselves are accurate.

CIA analysts do a much better job at linking their analysis to source reports, but only do a cursory examination of the accuracy of the source reporting. As a result, the inaccuracies littered throughout the raw intelligence can make their way into finished intelligence. Instituting a procedural mechanism to fact-check intelligence at both levels would provide an extra layer of assurance that the finished intelligence products were supplyingpolicy-makerswith accurate information.

The CIA also could improve its ability to warn policy-makers about issues that might negatively affect the national interestbyemphasizing alternative possibilities rather than a single assessment of the future. CIA analysts usually provide policy-makers with their assessment of the most likely outcome, but many times that single assessment turns out to be wrong because of factors the analysts did not incorporate into their analysis.

Intelligence analysts cannot be blamed for not addressing all the causal forces that affect international developments such as war, peace or the actions of foreign countries; there are so many that no one can take them all into account. However, the usual way other professions that deal with complex subjects — such as meteorology or epidemiology — forecast the future is by providing a range of possible outcomes and highlighting the factors that could change the outcomes. The CIA’s adoption of a scenario-based warning system highlighting multiple possible outcomes would more effectively capture analytic uncertainty than its current practice of providing a single assessment of the most likely outcome.

The CIA is better at telling policy-makers what it thinks it knows than communicating what it does not know. Spending more time and effort ensuring the accuracy of the intelligence product would only improve the CIA’s contribution to informing our national security policy-makers. In addition, relaying this information in a way that highlights uncertainty through formats that emphasize multiple possible outcomes would provide policy-makers with a better sense for what the CIA does not know. Overall, despite the inherent difficulties of the intelligence production process, the intelligence profession generally performs a good service in protecting and advancing the interests of the United States, but it has plenty of room for improvement.



Defense Capabilities and Management

General Accounting Office


Assessing Asperger’s syndrome

I appreciate the article that appeared in the Family Times on Feb. 15 describing Asperger’s syndrome (“Needing a world of order”). It is good to see something being done to get the word out on such things. However, I feel disgusted by the sense of despair conveyed. True, there is much area for worry and concern about such a pervasive social impairment, but your article only addressed that side of the issue. I suspect this condescending attitude toward people with Asperger’s syndrome as mildly freakish contributes directly toward the feelings of depression and inadequacy with which many of them deal. At least your article did not suggest any of the common stereotypes that suggest Asperger’s syndrome involves moral impairment, but the article failed to recognize a few things readers should know:

First, people with Asperger’s (some of whom I know) can be fascinating when one interacts with them. Also, they often are extraordinarily intelligent, with great potential in academic fields. It isn’t exceptional for them to become university professors, for instance. Obsession can easily translate into dedication and hard work. Finally, readers should remember that they are human beings with great intrinsic value.



Sign up for Daily Newsletters

Manage Newsletters

Copyright © 2021 The Washington Times, LLC. Click here for reprint permission.

Please read our comment policy before commenting.


Click to Read More and View Comments

Click to Hide