Yesterday, the Bureau of Justice Statistics (BJS) released a special report, Firearm Violence, 1993–2011. Not surprisingly, at least for those who follow crime statistics, the report shows that firearm homicides went down 39% between 1993 and 2011. The report also reconfirms many things that gun-rights supporters have been saying for decades: that less than 2% of prison inmates in 2004 bought their firearm from a “flea market or gun show,” and that “2% of state inmates and 3% of federal inmates were armed with a military-style semiautomatic or fully automatic firearm.”

Also not surprising is that very few people know about the dramatically reduced crime rate. Also released yesterday was a Pew study on Americans’ perceptions of the crime rate. Despite cutting the murder rate nearly in half in less than twenty years, only 12% of Americans believe that gun crime has dropped in the past two decades. Fifty-six percent believe it has increased, and 26% believe it stayed the same. This is not new. People often don’t realize how much better things are getting, and this fact can push public policy in misguided directions. 

Many have tried to explain this precipitous drop in crime, including one study that connected it to the decreased amount of lead in the environment. Whatever the cause, one thing is clear: there are about 50 million more guns in America now than in 1993 and crime did not go up.

Now, I will not oversell that statistic, not only because it does not prove the thesis “more guns, less crime,” but also because overselling statistics is a big problem in the gun control debate for both sides. For example, to take another statistic from the BJS report: the number of times per year people use guns to stop or curtail crime. 

Despite the fact that the BJS is quite good at some things, it is uniquely bad at measuring the level of defensive gun use (DGU) in America. And despite the fact that I can easily demonstrate this to anyone with even the slightest inclination to allow their minds to be changed, I am not optimistic that the gun controllers will listen. 

Gun controllers are constantly accusing gun-rights supporters of over-estimating the instances of DGU, and their primary source is the data from the National Crime Victimization Survey, which they rely on unquestioningly. The disparity between the BJS statistics and other studies is stark, as much as 30x. For example, yesterday’s BJS report claims that, between 2007–2011, crime victims used guns to stop or curtail crime 235,700 times. This aligns with the general tendency for the BJS to record between 60,000 and 100,000 DGUs per year. By contrast, Florida State’s award-winning criminologist Gary Kleck has found there may be as many as 2.5 million DGU instances per year. Gun-controllers almost always dismiss Kleck’s data as wildly inaccurate, if not NRA-funded propaganda (it is neither), and instead unquestioningly accept the BJS numbers. See, for example, this study by the Violence Policy Center, which simply regurgitates the BJS numbers, and this discussion of the VPC report at Mother Jones. This New York Times post on the VPC report sneeringly offers this observation on the disparity between Kleck’s and the BJS’s numbers:

Readers can judge for themselves whether the V.P.C. or the N.R.A. is likely to have better numbers. The V.P.C. used data from the National Crime Victimization Survey, conducted by the Bureau of Justice Statistics. The N.R.A.’s estimate is the result of a telephone survey conducted by a Florida State University criminologist.

I accept the Times’s invitation, and I will judge for myself:

Six Reasons to Distrust the NCVS on Defensive Gun Use

1) As you can see from the NCVS survey questions here, the survey is non-anonymous. Although respondents were given guarantees that the results will remain confidential, many respondents would certainly be uneasy about answering questions about gun use after giving their names and addresses. This is a well-known effect of non-anonymous surveys.

2) Respondents were told from the outset that the survey is conducted by the government, specifically the U.S. Census Bureau. Coupling this with the survey being non-anonymous and you have a perfect recipe for withholding information. This is all the more true if the respondent had reasonable concerns about whether an instance of DGU was legal, either because they themselves were not allowed to have a gun or to be carrying a gun (perhaps because of a felony conviction), because carrying a gun or having a gun is essentially illegal in their area (e.g., DC, NYC), or because of the obvious legal gray-area around the act of threatening someone with a gun, much less firing it at them. In essence, those who unquestionably trust the NCVS data on DGUs believe that the question, “hi, I’m from the government, please give me your name and address and tell me if you’ve used a gun to protect yourself in the past year” will yield accurate results.

3) Respondents were only asked about DGU after they had given the location of the crime. If the crime occurred outside the home, as did 78.1% of violent crimes between 2007–2011 (table 7), and some respondents were unauthorized to carry a gun outside the home, then they were being asked to admit to a crime.

4) But respondents were not just told that the U.S. Census Bureau is conducting the survey, they were told that the survey is sponsored by the Department of Justice. They were then asked to volunteer information about possibly illegal activities to the chief law enforcement agency of the United States.

5) Respondents were not directly asked about DGU because the NCVS is not primarily designed to uncover instances of DGU. They were first asked whether they have been victims of a crime and only then were they asked follow-up questions about the incident. But the NCVS under-reports crime, particularly domestic violence and rape, and if it under-reports crimes where DGUs are particularly common then it would of course under-report DGUs.

6) The follow-up questions are not straightforward enough to accurately capture instances of DGU. As you can see from the follow-up questionnaire given to those who report a crime (page 12), respondents must first have answered “yes” to the question, “Did you do anything with the idea of protecting YOURSELF or your PROPERTY while the incident was going on?” before they were asked about DGU. They were then asked to volunteer information about a DGU rather than being put in a position of having to lie in order to deny a DGU.

That’s what the government did. Let’s see how “the result[s] of a telephone survey conducted by a Florida State University criminologist (Gary Kleck)” were collected:

We use the most anonymous possible national survey format, the anonymous random digit dialed telephone survey. We did not know the identities of those who were interviewed, and made this fact clear to the Rs [Respondants]. We interviewed a large nationally representative sample covering all adults, age eighteen and over, in the lower forty-eight states and living in households with telephones. We asked DGU questions of all Rs in our sample, asking them separately about both their own DGU experiences and those of other members of their households. We used both a five year recall period and a one year recall period. We inquired about uses of both handguns and other types of guns, and excluded occupational uses of guns and uses against animals. Finally, we asked a long series of detailed questions designed to establish exactly what Rs did with their guns; for example, if they had confronted other humans, and how had each DGU connected to a specific crime or crimes.

Interviews were monitored at random by survey supervisors. All interviews in which an alleged DGU was reported by the R were validated by supervisors with call-backs, along with a 20% random sample of all other interviews.

Questions about the details of DGU incidents permitted us to establish whether a given DGU met all of the following qualifications for an incident to be treated as a genuine DGU: (1) the incident involved defensive action against a human rather than an animal, but not in connection with police, military, or security guard duties; (2) the incident involved actual contact with a person, rather than merely investigating suspicious circumstances, etc.; (3) the defender could state a specific crime which he thought was being committed at the time of the incident; (4) the gun was actually used in some way–at a minimum it had to be used as part of a threat against a person, either by verbally referring to the gun (e.g., “get away–I’ve got a gun”) or by pointing it at an adversary. We made no effort to assess either the lawfulness or morality of the Rs’ defensive actions.

An additional step was taken to minimize the possibility of DGU frequency being overstated. The senior author went through interview sheets on every one of the interviews in which a DGU was reported, looking for any indication that the incident might not be genuine. A case would be coded as questionable if even just one of four problems appeared: (1) it was not clear whether the R actually confronted any adversary he saw; (2) the R was a police officer, member of the military or a security guard, and thus might have been reporting, despite instructions, an incident which occurred as part of his occupational duties; (3) the interviewer did not properly record exactly what the R had done with the gun, so it was possible that he had not used it in any meaningful way; or (4) the R did not state or the interviewer did not record a specific crime that the R thought was being committed against him at the time of the incident. There were a total of twenty-six cases where at least one of these problematic indications was present. It should be emphasized that we do not know that these cases were not genuine DGUs; we only mean to indicate that we do not have as high a degree of confidence on the matter as with the rest of the cases designated as DGUs. Estimates using all of the DGU cases are labelled herein as “A” estimates, while the more conservative estimates based only on cases devoid of any problematic indications are labelled “B” estimates.

Kleck’s study (which you can read here) was done in 1993, so there is good reason to believe that the instances of DGU have gone down with the crime rate. Also, Kleck’s study, like all studies, is far from ironclad. But I’ll leave that question to the reader: which study is likely to be more reliable?

(For more on defensive gun use, see Cato’s study Tough Targets. You can also check out interactive map on defensive gun use.)