Sociological Realities and Econometric Illusions
Skeptical Enquirer Magazine, July 2004
Does executing murderers cut the homicide rate or not? Comparative studies show there is no effect. Econometric models, in contrast, show a mixture of results. Why the difference? And which is the more reliable method?
By Ted Goertzel
I have inquired for most of my adult life about studies that might show that the death penalty is a deterrent, and I have not seen any research that would substantiate that point.
-- Attorney General Janet Reno, January 20, 2000
All of the scientifically valid statistical studies—those that examine a period of years, and control for national trends—consistently show that capital punishment is a substantial deterrent.
-- Senator Orrin Hatch, October 16, 2002
It happens all too often. Each side in a policy debate quotes studies that support its point of view and denigrates those from the other side. The result is often that research evidence is not taken seriously by either side. This has led some researchers, especially in the social sciences, to throw up their hands in dismay and give up studying controversial topics. But why bother doing social science research at all if it is impossible to obtain accurate and trustworthy information about issues that matter to people?
There are some questions that social scientists should be able to answer. Either executing people cuts the homicide rate or it does not. Or perhaps it does under certain conditions and not others. In any case, the data are readily available and researchers should be able to answer the question. Of course, this would not resolve the ethical issues surrounding the question, but that is another matter.
So who is right, Janet Reno or Orrin Hatch? And why can they not at least agree on what the data show? The problem is that each of them refers to bodies of research using different research methods. Janet Reno’s statement correctly describes the results of studies that compare homicide trends in states and countries that practice capital punishment with those that do not. These studies consistently show that capital punishment has no effect on homicide rates. Orrin Hatch refers to studies that use econometric modeling. He is wrong, however, in stating that these studies all find that capital punishment deters homicide. In fact, some of them find a deterrent effect and some do not.
But this is not a matter of taste. It cannot be that capital punishment deters homicide for comparative researchers but not for econometricians. In fact, the comparative method has produced valid, useful, and consistent findings, while econometrics has failed in this and every similar area of research.
The first of the comparative studies of capital punishment was done by Thorsten Sellin in 1959. Sellin was a sociologist at the University of Pennsylvania and one of the pioneers of scientific criminology. He was a prime mover in setting up the government agencies that collect statistics on crime. His method involved two steps: “First, a comprehensive view of the subject which incorporated historical, sociological, psychological, and legal factors into the analysis in addition to the development of analytical models; and second, the establishment and utilization of statistics in the evaluation of crime” (Toccafundi 1996).
Sellin applied his combination of qualitative and quantitative methods in an exhaustive study of capital punishment in American states. He used every scrap of data that was available, together with his knowledge of the history, economy, and social structure of each state. He compared states to other states and examined changes in states over time. Every comparison he made led him to the “inevitable conclusion . . . that executions have no discernable effect on homicide rates” (Sellin 1959, 34).
Sellin’s work has been replicated time and time again, as new data have become available, and all of the replications have confirmed his finding that capital punishment does not deter homicide (see Bailey and Peterson 1997, and Zimring and Hawkins 1986). These studies are an outstanding example of what statistician David Freedman (1991) calls “shoe leather” social research. The hard work is collecting the best available data, both quantitative and qualitative. Once the statistical data are collected, the analysis consists largely in displaying them in tables, graphs, and charts which are then interpreted in light of qualitative knowledge of the states in question. This research can be understood by people with only modest statistical background. This allows consumers of the research to make their own interpretations, drawing on their qualitative knowledge of the states in question.
Figure 1: Homicide rates per 100,000 population in Texas, New York, and California.
Figure 1 is an example of the kind of chart Sellin prepared, using recent data. The graph compares homicide rates per 100,000 population in Texas, New York, and California. From 1982 to 2002, Texas executed 239 prisoners, California ten, and New York none. The trends in homicide statistics are very similar in all three states, all of which follow national trends. These states were chosen arbitrarily, but data for other states are readily available. If you prefer to compare Texas to Oklahoma, Arkansas, or New Mexico, the data are readily available in back issues of the Statistical Abstract of the United States and Uniform Crime Reports. The results will be much the same.
Hundreds of comparisons of this sort have been made, and they consistently show that the death penalty has no effect. There have also been international comparative studies. Archer and Gartner (1984) examined fourteen countries that abolished the death penalty and found that abolition did not cause an increase in homicide rates. This research has been convincing to most criminologists (Radelet and Akers n.d.; Fessenden 2000), which is why Janet Reno was told that there was no valid research linking capital punishment to homicide rates.
The studies that Orrin Hatch referred to use a very different methodology: econometrics, also known as multiple regression modeling, structural equation modeling, or path analysis. This involves constructing complex mathematical models on the assumption that the models mirror what happens in the real world. As I argued in a previous Skeptical Inquirer article (Goertzel 2002), this method has consistently failed to offer reliable and valid results in studies of social problems where the data are very limited. Its most successful use is in making predictions in areas where there is a large flow of data for testing. The econometric literature on capital punishment has been carefully reviewed by several prominent economists and found wanting. There is simply too little data and too many ways to manipulate it. In one careful review, McManus (1985, 417) found that: “there is much uncertainty as to the ‘correct’ empirical model that should be used to draw inferences, and each researcher typically tries dozens, perhaps hundreds, of specifications before selecting one or a few to report. Usually, and understandably, the ones selected for publication are those that make the strongest case for the researcher’s prior hypothesis.”
Models that find deterrence effects of capital punishment often rely on rather bizarre specifications. In a rigorous and comprehensive review Cameron (1994, 214) observed that, “What emerges most strongly from this review is that obtaining a significant deterrent effect of executions seems to depend on adding a set of data with no executions to the time series and including an executing/non-executing dummy in the cross-section analysis . . . there is no clear justification for the latter practice.”
In less technical language, the researchers included a set of years when there were no executions, then introduced a control variable to eliminate the nonexistent variance. The other day upon the stair, they saw some variance that wasn’t there. It wasn’t there again today, thank goodness their model scared it away. Not all the studies rely on this particular maneuver, but they all depend on techniques that demand too much from the available data.
Since there are so many ways to model inadequate data, McManus (1985, 425) was able to show that researchers whose prior beliefs led them to structure their models in different ways would obtain predictable conclusions: “The data analyzed are not sufficiently strong to lead researchers with different prior beliefs to reach a consensus regarding the deterrent effects of capital punishment. Right-winger, rational-maximizer, and eye-for-an-eye researchers will infer that punishment deters would-be murderers, but bleeding-heart and crime-of-passion researchers will infer that there is no significant deterrent effect.”The Mythical World of Ceteris Paribus
Econometricians inhabit the mythical land of Ceteris Paribus, a place where everything is constant except the variables they choose to write about. Ceteris Paribus has much in common with the mythical world of Flatland in Edwin Abbot’s (1884) classic fairy tale. In Flatland everything moves along straight lines, flat plains, or rectangular boxes. In Flatland, statistical averages become mathematical laws. For example, it is true that, on the average, tall people weigh more than short people. But, in the real world, not every tall person weighs more than a shorter one. In Flatland knowing someone’s height would be enough to tell you their precise weight, because both vary only on a straight line. In Flatland, if you plotted height and weight on a graph with height on one axis and weight on the other, all the points would fall on a straight line.
Of course, econometricians know that they don’t live in Flatland. But the mathematics works much better when they pretend they do. So they adjust the data in one way or another to make it straighter (often by converting it to logarithms). Then they qualify their remarks, saying “capital punishment deters homicide, ceteris paribus.” But when the real-world data diverge greatly from the straight lines of Flatland, this can lead to bizarre results.
Figure 2: Anscombe’s Quartet (by J. Randall Flannigan)
Statistician Francis Anscombe (1973) demonstrated how bizarre the Flatland assumption can be. He plotted four graphs that have become known as Anscombe’s Quartet. Each of the graphs shows the relationship between two variables. The graphs are very different, but for a resident of Flatland they are all the same. If we approximate them with a straight line (following a “linear regression equation”) the lines are all the same (figure 2). Only the first of Anscombe’s four graphs is a reasonable candidate for a linear regression analysis, because a straight line is a reasonable approximation for the underlying pattern.
Figure 3: Executions and murder rates in the United States.
The data on capital punishment and homicide, when plotted in figure 3, look a lot like Anscombe’s fourth quartet. Most of the states had no executions at all. One state, Texas, accounts for forty of the eighty-five executions in the year shown (the patterns for other years are quite similar). An exceptional case or “outlier” of this dimension completely dominates a multiple regression analysis. Any regression study will be primarily a comparison of Texas with everywhere else. Multiple regression is simply inappropriate with this data, no matter how hard the analyst tries to force the data into a linear pattern.
Unfortunately, econometricians continue to use multiple regression on capital punishment data and to generate results that are cited in Congressional hearings. In recent examples, Mocan and Gittings (2001) concluded that each execution decreases the number of homicides by five or six while Dezhbaksh, Rubin, and Shepherd (2002) argued that each execution deters eighteen murders. Cloninger and Marchesini (2001) published a study finding that the Texas moratorium from March 1996 to April 1997 increased homicide rates, even though no increase can be seen in the graph (figure 1). The moratorium simply increased homicide in comparison to what their econometric model said it would have otherwise been. Of all the econometric myths, the wildest is this: We know what would have been.
Cloninger and Marchesini concede that “studies such as the present one that rely on inductive statistical analysis cannot prove a given hypothesis correct.” However, they argue that when a large number of such studies give the same result, this provides “robust evidence” which “causes any neutral observer pause.” But if McManus is correct that econometricians are likely to specify models to fit their preconceptions, then if many of them reach the same conclusion it may just mean that they have the same bias. Actually, there are a variety of biases among econometricians, which is why there are almost as many on one side as on the other of this issue. In response to Ehrlich’s (1975) initial econometric study, other econometricians using the same data included Yunker (1976), who found a stronger deterrent effect than Ehrlich, and Cloninger (1977), who supported his findings. But Bowers and Pierce (1975), Passel and Taylor (1977), and Hoenack and Weiler (1980) found no deterrence at all.
Econometricians often dismiss the kind of comparative research that Thorsten Sellin did as crude and unsophisticated when compared to their use of complex mathematical formulas. But mathematical complexity does not make for good social science. The goal of multiple regression is to convert messy sociological realities into math problems that can be resolved with the certainty of mathematical proof. Econometricians believe they can control for the myriad variables that affect homicide rates, just as a chemist eliminates impurities to see how two chemicals interact in their pure form. But they cannot convert the real world into a Flatland, so they use statistical adjustments to compensate. With these adjustments, they claim to answer the Ceteris Paribus question: If everything else were equal, what would the relationship between capital punishment and homicide be?
It would be handy for social scientists if we lived in a Flatland where everything else was equal and questions could be answered with a few calculations. But multivariate statistical analysis does not answer real-world questions such as, “does Texas, with a high execution rate, have a lower homicide rate than similar states?” or “did the homicide rate go down when Texas began executing people, compared to trends in other states that did not?” Instead, it answers the question, “If we use the latest, most sophisticated statistical methods to control for extraneous variables, can we say that the death penalty deters homicide rates other things being equal?” After decades of effort by many diligent researchers, we now know the answer to this question: There are many ways to adjust things statistically, and the answer will depend on which one is chosen. We also know that of the many possible ways to specify a regression model, each researcher is likely to prefer one that will give results consistent with his or her predispositions.
It is time to abandon the illusion that mathematics can convert the real world into the mythical land of Ceteris Paribus. Social science can provide valid and reliable results with methods that present the data with as little statistical manipulation as possible and interpret it in light of the best qualitative information available. The value of this research is shown by its success in demonstrating that capital punishment has not deterred homicide.
Abbot, Edwin. 1884. Flatland: A romance of many dimensions. Accessed on January 29, 2004, at: www.alcyone.com/max/lit/flatland/.
Anscombe, Francis. 1973. Graphs in statistical analysis. American Statistician 27, 17–21.
Archer, Dane, and Rosemary Gartner. 1984. Homicide and the death penalty: A cross-national test of a deterrence hypothesis. In Archer and Gartner, Violence and Crime in Cross-National Perspective, New Haven: Yale University Press.
Bailey, William, and Ruth Peterson. 1997. Murder, capital punishment, and deterrence: A review of the literature. In Hugo Bedau, ed., The Death Penalty In America: Current Controversies. New York: Oxford University Press.
Bowers, W.J., and J.L. Pierce. 1975. The illusion of deterrence in Isaac Ehrlich’s work on capital punishment. Yale Law Journal 85: 187–208.
Cameron, Samuel. 1994. A review of the econometric evidence on the affects of capital punishment. Journal of Socio-Economics 23: 197–214.
Cloninger, Dale. 1977. Deterrence and the death penalty: A cross-sectional analysis. Journal of Behavioral Economics 6, 87–107.
Dezhbaksh, Hashem, Paul Robin, and Joanna Shepherd. 2002. Does capital punishment have a deterrent effect? New evidence from post-moratorium panel data. American Law and Economics Review 5(2): 344–376. Accessed on January 12, 2004, from: http://aler.oupjournals.org/cgi/content/ abstract/5/2/344.
Ehrlich, Isaac. 1975. The deterrent effect of capital punishment: A question of life and death. American Economic Review 65: 397–417.
Fessenden, Ford. 2000. Deadly statistics: A survey of crime and punishment. The New York Times September 22, 2000. Accessed on January 20, 2004 from: www.nytimes.com
Freedman, David. 1991. Statistical models and shoe leather. Sociological Methodology 21: 291–313.
Goertzel, Ted. 2002. Myths of murder and multiple regression. Skeptical Inquirer 26(1): 19–23.
Hatch, Orrin. 2002. Minority views, appended to the “report” of the Committee on the Judiciary of the United States Senate on The Innocence Protection Act of 2002, 107th Congress, Second Session, Report 107-315, Calendar No 731. Accessed on January 12, 2004, from: http://frwebgate.access.gpo.gov/
Hoenack, Stephen, and William Weiler. 1980. A structural model of murder behavior and the criminal justice system. American Economic Review 70, 327–41.
McManus, Walter. 1985. Estimates of the deterrent effect of capital punishment: The importance of the researcher’s prior beliefs. Journal of Political Economy 93: 417–425.
Mocan, Naci, and Kaj Gittings. 2001. Pardons, executions and homicide. Working Paper 8639, National Bureau of Economic Research. Accessed on January 12, 2004, from: www.nber.org/papers/w8639.
Passell, Peter, and John Taylor. 1977. The deterrent effect of capital punishment: Another view. American Economic Review 67, 445.
Radelet, Michael, and Ronald Akers. (No date). Deterrence and the death penalty: The view of experts. accessed on January 23, 2004, from: http://sun.soci.niu.edu/.
Reno, Janet. 2002. Quoted in Mocan and Gittings 2001.
Sellin, Thorsten. 1959. The Death Penalty. American Law Institute, Philadelphia.
Toccafundi, David. 1996. The Sellin Collection at Penn. Accessed on January 24, 2004, at: http://www.library.upenn.edu/exhibits/pennhistory/library/sellin.html.
Yunker, James A. 1976. Is the death penalty a deterrent to homicide? Some time series evidence. Journal of Behavioral Economics Vol. 5, 45–81.
Zimring, Franklin, and Gordon Hawkins. 1986. Capital Punishment and the American Agenda. Cambridge: Cambridge University Press.About the Author Ted Goertzel is a professor of sociology, Sociology Department, Rutgers University, Camden, NJ 08102. E-mail: goertzel@camden. rutgers.edu.