A recent Wall Street Journal article brought to light a report from the Department of Energy on the origin of the COVID pandemic. The rating attached to the report in the WSJ article highlights the importance of understanding the intelligence confidence scale. The Department of Energy report concluded that the pandemic most likely arose from a laboratory leak with a “low confidence” rating attached to the findings. Meanwhile, the FBI came to the same conclusion but with “moderate confidence” and still holds to this view.
The US Intelligence Community relies on a confidence scale to assess and communicate the level of certainty in their intelligence reports. This scale is used to rate the level of confidence that an analyst has in their judgements, including those already qualified by probability terms such as “very unlikely” or “almost certainly”. The use of the confidence scale is crucial in helping intelligence report consumers make informed decisions, but it is often misunderstood or misapplied by intelligence practitioners.
One such scale that is used within the Intelligence Community is the DIA Tradecraft Note 01-15, which is a 5-point scale. This scale references a 2013 memo from the Office of the Director of National Intelligence and stresses the distinction between estimative probability and analytic confidence. Confidence is based on various factors such as information credibility, source reliability, and the persuasiveness of the analyst’s arguments, while estimative probability is used to communicate likelihood.
Here’s a breakdown of each level of confidence according to the DIA Tradecraft Note 01-15:
- Complete Confidence:
- Totally reliable and corroborated information with no assumptions
- Clear, undisputed reasoning
- Contextual/historical or other knowledge base foundation to support this level
- High Confidence:
- Well-corroborated information from proven sources
- Extensive databases and/or historical understanding of the issue
- Low potential for deception
- Assumptions used to fill gaps are not critical to the analysis
- Reasoning dominated by strong logical inferences
- Moderate Confidence:
- Partially corroborated information from good sources
- Some databases and/or historical understanding of the issue
- Moderate potential for deception
- Assumptions potentially critical to the analysis are used to fill gaps
- Reasoning with a mixture of strong and weak inferences
- Low Confidence:
- Uncorroborated information from good or marginal sources
- Minimal databases and/or historical understanding of the issue
- High potential for deception
- Key assumptions critical to the analysis are used to fill gaps
- Reasoning dominated by weak inferences
- No Confidence:
- No direct information to support the assessment
- Products with no confidence should only be used for exploratory analysis
The intelligence confidence scale is a useful tool for communicating the level of certainty in intelligence reports, but it is important to understand the nuances of each level of confidence and how it is applied. As highlighted in the case of the Department of Energy report, different agencies may arrive at different conclusions for different reasons, despite using the same confidence scale.
A “Low Confidence” rating was attached to the DOE report containing the finding that the COVID Pandemic most likely arose from a laboratory leak. These are the specific factors that merit a “Low Confidence” rating:
- Uncorroborated information from good or marginal sources
- Minimal databases and/or historical understanding of the issue
- High potential for deception
- Key assumptions critical to the analysis are used to fill gaps
- Reasoning dominated by weak inferences
A rating of “Moderate Confidence” was attached to the FBI report containing the finding that the COVID pandemic most likely arose from a laboratory leak. These are the specific factors that merit a “Moderate Confidence” rating:
- Partially corroborated information from good sources
- Some databases and/or historical understanding of the issue
- Moderate potential for deception
- Assumptions potentially critical to the analysis are used to fill gaps
- Reasoning with a mixture of strong and weak inferences
Beyond what speculation is already flowing through various media channels, there is one item that is worth taking note of, in the case of both reports related to the origins of the COVID pandemic, the DOE report and the FBI report, both received a higher confidence rating than the lowest.
- No Confidence:
- No direct information to support the assessment
- Products with no confidence should only be used for exploratory analysis