Larry Johnson and the Numbers Game (Updated)

Demosophist

I had a brief debate with Larry Johnson on the Jawa Report last week, that was recently referenced by an article in Mother Jones Blog. I expanded on the issue briefly here. To summarize, my position has basically been that:

1. The inference that State is somehow avoiding empirical evidence related to its policies is not justified by what we know and what Larry has reported. It's more likely that they are just tired of confounding policy with a report whose only real value is in a fruitless advocacy conflict; and

2. That the increase in attacks, even if specified and counted correctly, is not a meaningful indicator of whether we are "making progress" in the War on Terror, though Larry insists the data demand some negative inference.

In addition, I also suggested:

3. That if the method used to identify a terrorist event were mis-specified, and that portion affected by the mis-specification was the one that increased during the period, or that more incidents are simply being reported (which is a commonplace issue with crime statistics) then we may not actually be looking at an increase. (It's unlikely that this has seriously distorted the numbers, but we haven't checked very carefully.)

But even if we concede that the numbers are largely correct and consistent with previous counts, there are numerous reasons why it may not be qualitatively a very good metric. In fact it appears to even include thwarted attacks, if they happen to kill or injure a small number of people. Should these thwarted attacks really be counted towards a lack of progress?

But regardless of whether we have a qualitatively good metric (and we almost certainly don't), we have probably mis-specified terrorist attacks relevant to the real status of the war, and that's the central issue.

If Armed Liberal is correct (and it appears he is) that 30% of the attacks were in Iraq, and Andrew Cochran is correct that almost half occurred in Kashmir (and that's not even in dispute), then what we have are two localized conflicts accounting for more than 75% of the attacks in 2004, both of which are being lost by the terrorists. (Pakistan and India are close to agreement over Kashmir.)

Just how does this translate into a "lack of progress?" Moreover, is Larry's conclusion that "... the news is not good for U.S. efforts to contain and reduce the threat of international terrorism" even relevant? I submit that the fact that he's a specialist may be clouding his perspective. The real issue is whether we're winning. Wouldn't that be the relevant "good news?" And what reason do we really have to doubt that we're winning?

Futhermore, consider that if we are indeed winning against a terrorist movement, that's just about the first time it has ever happened! Up until recently "asymmetric warfare" of the sort that targets civilians as a deliberate strategy, has been an enormously successful strategy. Thus, it's hardly surprising that a winning strategy may not look much like a walk in the park.

If we'd used the same sort of metric in WWII to gauge success or failure we'd have observed higher and higher losses with each passing year and a big jump at the Battle of the Bulge and Iwo Jima, which were when we actually began to close on the enemy. It was certainly "not good news" that we lost a lot of fine men on Iwo Jima, but what does that have to do with the crux of the matter? Would it have been better to leave the strategically important island in the hands of the Japanese?

Incidentally Princeton has a series of webcasts of its ongoing Colloquium on Public and International Affairs. If readers go to that link and scroll about halfway down the page there's a synopsis and several links to the video of a panel discussion entitled Measuring Success in Combating Terrorism. The speakers in that discussion include Larry Johnson, but they also include Raphael Perl from the Congressional Research Service and Peter Probst, a former CIA officer. It's an excellent panel, and well worth the time spent to view it.

My feeling is that if Larry Johnson had paid more attention to the other speakers on that panel, and especially Raphael Perl, we wouldn't be having this debate. Perl is especially eloquent about the fact that we don't currently have any metrics that can tell us not only whether, but the degree to which, we're winning. And we need them. To that extent I agree with Johnson, that the numbers are important. But I don't agree that these numbers tell us much.

One of the biggest deficiencies in our recording and analysis is that we have no measure of the "quality" of the attacks, nor do we even bother to sort them by scale. In fact, we often don't even distinguish between attacks that are carried out successfully and those that were pre-empted! But the greatest deficiency in our empirical analysis of the War Against Totalitarianism 3.0 is that in a war which is less about battles and casualties than intangible perceptions and attitudes, we don't bother to pay much attention to context.
Hence, we can look at an increase in the overall number of attacks and completely miss the salient fact that a vast majority of those incidents are taking place in locations where the terrorists are losing the war of ideas, and where the political situation is being resolved against their interests.

Update: Regarding the first part of Larry Johnson's Why the Numbers Matter. (I'll try to get to the rest when I have time.)

Q: What is the significance of the fact that in 1986 State was placed "in charge of coordinating the efforts of CIA, DOD, and FBI efforts [sic] to track and deal with terrorism," and the apparent inconsistency of Phil Zelikow's statement that it is the NCTC, rather than State, that was tasked with "analysis and integration of intelligence (data) on terrorism or counterterrorism" and would act as the "shared knowledge bank" on such data and analysis?

A: None. One is a managerial task while the other is a technical one.

Q: What is the significance of Phil Zelikow's apparent misstatement that State "traditionally compiled the data," when all it really did was the managerial function of coordination? The statement simply uses the word "compiled" instead of "published" in case you missed that.

A: It might be regarded as somewhat imprecise to substitute "compiled" for "published," but is it a strange or even erroneous conflation? In fact, one of the meanings in Webster's for "compile" is "to compose, out of materials from other documents." In fact that's the first meaning, and therefore the oldest! And isn't this essentially what they did? Only a specialist would tend to immediately conclude that the speaker was obfuscating the technical function of compiling raw data with that of publishing a finished report. Basically there's simply no case that Zelikow or Brennan misrepresented anything at all, so we don't even need to deal with the problem of asking whether a misrepresentation (that didn't happen) was intentional or inadvertent.

Additionally Larry contends that the move to compile and publish the data at the NCTC is "stovepiping" of the sort that the 9/11 report cautions against. If this is true then Zelikow and Brennan are simply lying about the statute-mandated technical function of the NCTC which is consistent with this "stove-piping." Is it plausible that they'd lie about that? Johnson doesn't even claim that they did. So what's the sense of his argument? Why is he even making it?

I'm not sure, but it seems that he's alarmed that the NCTC has usurped the task of "compiling the report" (in the publishing sense). What this has to do with stove-piping I don't know. Stove-piping is a process by which data, at a relatively raw level of development, is usurped or diverted to be interpreted by analysts at the executive level, presumably without the proper training. So what does this have to do with usurping the compilation of the report? Even if this was happening in some illegitimate sense, it's not stove-piping, because it's a matter of the technical people interpreting the data they're responsible for analyzing. They're doing what they're supposed to do.

And it may serve to point out that although one could use the compositional function of pulling together data to "back-coordinate" the sources of those data, which would be instrumental to the managerial function, it's but one way to skin that cat. Is NCTC usurping a managerial function that ought to be performed by state. (Note: Although this isn't stove-piping it would stlill be problematic.) Well, provided NCTC has the editors, writers, and technical advisors to make sense of the data they could not only do it, but in view of the increasing complexity of the mission, they'd probably do it better. As long as they aren't coordinating or formulating policy, they aren't doing anything untoward.

Q: What is the significance of the claim (fact?) that when the CIA shifted responsibility for counting terrorist incidents to the Terrorist Threat Integration Center (TTIC) (an interagency body ostensibly part of State's coordinative mission, established in 1986), the organization had only three part-time personnel assigned to the task?

A: Well for one thing, if true, it would mean that until the staff was increased, the number of incidents may have been undercounted. In that case the numbers in 2003 would not be comparable to the numbers in 2004, so its not valid to discuss a "trend." (Note that this was one of my original objections to taking these data too seriously.) But leading us to that conclusion would also count as an incentive for Brennan to lie about the number and status of people assigned to that task. Johnson, however, assures us that Brennan is telling the truth: "This process [the collaborative data checking between the NCTC and State] broke down when the responsibility for doing this [counting terrorist incidents] was shifted from CTC and put under Mr. Brennan’s stewardship at the Terrorist Threat Integration Center in late 2003."

Are you dizzy yet? To regroup: the NCTC starts with responsibility for the technical function of counting the number of terrorist incidents, as part of its responsibility for the "analysis and integration of intelligence" function. This technical function was moved to the interagency organization, the TTIC, in 2003 and quickly fumbled. John Brennan, however, is the head of both the TTIC and the NCTC, so unless these were serial appointments (which is something I don't know) he's simultaneously the head of the organization that supposedly did a stellar job pre-2003 and the organization that fumbled the ball in 2003. The TTIC was supposedly under the purview of State according to its 1986 directive and is headed by the same person as the NCTC: John Brennan. At this point I seem to have dropped my compass, and am badly in need of a GPS fix. The TTIC seems to have both a technical and managerial/executive function, which may explain why it was a bit dysfunctional.

And unfortunately (or fortunately, depending on your point of view) the Homeland Security Act has stepped in to blur these boundaries even further. Is it any wonder that the TTIC's responsibility has, again, been superceded by the NCTC (as the TTIC's website announces) where it resided prior to 2003, and where even Larry Johnson claims a good job was done?

Now it's true that Larry chides Brennan's management of the NCTC a bit for requiring 10 employees to keep track of 650 terrorist incidents in 2004, but the issue isn't so much whether they were overstaffed as whether the TTIC was understaffed in 2003. Larry has hold of the wrong end of that telescope. And the bottom line is that it's quite possible they were understaffed. At least, Brennan is accurately representing the staffing they had even though that fact tends to reflect badly on him. Anyway, Johnson observes that the magnitude of the number of terrorist incidents is comparable to counts produced by other sources, so they probably aren't wildly off the mark. If Brennan was attempting to give the impression that the 2003 count was low that's probably misleading, but that lets him off the hook for incompetence in running the TTIC understaffed. And the fact that the numbers are comparable doen't really rescue the metric itself from irrelevance. So the number increased. What does that mean? It means we haven't yet defeated the enemy. It doen't mean we aren't winning.

If you see something irregular either in the decision to move responsibility back to the NCTC or anything else, please tell me. As far as I can tell it all makes perfect sense. If there was an organizational or managerial error it was in moving responsibility for the technical function to the TTIC to begin with. But I imagine that was just one of those "coordinative experiments" that didn't work out.

We certainly need to see better performance from these guys, and we also need to see them build some more useful metrics for keeping track of the WoT and how we're fairing in that conflict. But I don't see anything that indicts this administration, or that suggests we aren't making progress in the WoT. Let's keep our focus, shall we?

(Cross-posted by Demosophist to Demosophia and The Jawa Report)

Launched by Demosophist at April 29, 2005 06:40 PM