Category Archives: Methodology

Terrorism Data Remains a Mess

Posted on by
3

I’ve recently been trying to get a handle on terrorism trends in Pakistan, and in that process have been reminded of the problems in terrorism datasets. Based on extant data, I could tell you two stories about Pakistan: terrorism in Pakistan is either getting progressively worse or has gotten considerably better since 2009.

Here is the basic trend for terrorism in Pakistan using the National Consortium for the Study of Terrorism and Responses to Terrorism (START) Global Terrorism Database (GTD).

Image

Compare that with the trend for all terrorism using the National Counterterrorism Center’s World Incident Tracking System. (Through some process opaque to me, the U.S. government decided to stop producing this data series. Or they likely continue to produce it, but just don’t provide it to the outside world. And ignore the redline, which is just the mean across the years shown.)

Image

If you believe GTD, you should be really worried about Pakistan. If you believe NCTC, we may have turned a corner. Get your “Mission Accomplished” banner ready.

The evidence is a little more consistent when looking at just suicide terrorism. Here is the data from the University of Chicago’s Project on Security and Terrorism (CPOST). Sadly, this data hasn’t been updated since October 2011, suggesting they ran out of funding, and also meaning the 2011 data below is incomplete. But, the data suggest a strong peak at 2009, and then sizeable decreases since then, even with the incomplete 2011 data.

Image

Finally, compare that with the GTD data for just suicide attacks. Strong peak in 2007, and then a more modest decline since then.

Image

I don’t have much in the way of conclusions, but despite how many asterisks are found in coefficient charts, we should be skeptical of terrorism findings that are not robust across datasets, and given the discrepancies across data sets, such robustness may be unlikely. The long twilight struggle for cumulative knowledge continues.

Nuclear Weapons and War

Posted on by
1

I wanted to flag a soon-to-be-published article by my colleagues at MIT, Mark Bell and Nicholas Miller, looking at whether and how nuclear weapons possession affects conflict.

The paper is interesting on both substantive and methodological grounds. Substantively, they find no statistically discernable difference in the conflict propensity of states with offsetting nuclear arsenals. This is true even at low levels of conflict. These two findings combined mean they find insufficient evidence to support the existence of a “stability-instability” paradox where the presence of nuclear weapons both deters full-scale war while also increasing the likelihood of lower levels of violence.

I think the methodological point they make is perhaps more important, at least for political science practitioners. There was an established paper in the field that found empirical support for the stability-instability paradox, a paper by Rauchhaus (2009). Rauchhaus made a few mistakes, and helpfully for those of us playing at home, mistakes that we probably could imagine making ourselves.

Empirically, he coded the 1999 Kargil conflict between India and Pakistan as a non-war. The number of casualties in that conflict is somewhat in dispute, and alas for us, it is either immediately below or immediately above the 1,000 battle deaths standard that has come to define “war” in political science. Rauchhaus generally relied on the Correlates of War coding of conflict, except in the Kargil case, which means that Rauchhaus accepted the Correlates of War coding for many wars with fewer casualties than Kargil. The empirical finding that nuclear dyads are less likely to fight wars is entirely dependent on whether or not Kargil is coded as a war. If it is coded as a war, then there is no statistically significant difference between nuclear dyads and their non-nuclear counterparts. Though Bell and Miller do not mention it, Kargil has the potential to play a similar spoiler role in the deterministic-variant of the democratic peace literature, and so people would be well advised to always pay attention to Kargil in their dataset if war is an important IV or DV. Montgomery and Sagan (2009) had flagged this a while ago, but it’s not clear to me it has fully sunk in.

The other error is also one that could be made more generally, and so should be of interest to political scientists uninterested in war or nuclear weapons. He used a canned package in Stata called xtgee, which estimates a logit generalized estimating equation (GEE). Exciting stuff, no? The problem is, that when Kargil is coded as a non-war, then there are no wars between nuclear dyads, which creates “separation” in the data. Nuclear weapons predict non-war perfectly. This should lead to non-identification in GEE, or logit, or probit models. The computer should yell at you in such instances. In this case, the xtgee command in Stata erroneously allowed for a coefficient estimate to be produced, and hence Rauchhaus found what he found. Rauchhaus might have realized his results were fishy if he had produced a table of relative risk tables instead of just reporting coefficient estimates in the article. If he had done so, he would have realized his coefficient suggested non-nuclear dyads were 2.7 million times more likely to go to war than nuclear dyads. Bell and Miller use an estimator developed by Firth that allows for parameter estimation even with separation in the data (a downloadable firthlogit package is available for Stata users).

Separation is a fairly common problem, particularly for small datasets or datasets with rare events in them that are dichotomously coded. It should be underlined that a common response of statistical programs is to drop variables with separation, which permits a computational solution, but probably biases the results on the remaining variables. This webpage by UCLA helpfully walks through how different statistical packages handle the problem. Others who are methodologically smarter than me have convinced me to always fit a linear probability model onto my data as a robustness check. Linear probability models also do not suffer from separation. And if the coefficient estimates are radically off, you should just be prepared to defend your estimator choice very strongly.

Survivorship Bias, Sample Sizes, and the Oregon Medicaid Study

Posted on by
4

I think most coverage of the Oregon Medicaid Study [gated] has been bad. Very bad. I wanted to flag one way that it has been especially bad.

We don’t do very much U.S. domestic politics on the Smoke-Filled Room but I think the broader methodological issues are worth highlighting. So, for those that don’t obsessively follow wonkish U.S. policy debates, a bit of background. When Oregon expanded Medicaid coverage a few years back, it did so via a lottery. That allowed researchers to compare outcomes between those who received Medicaid and those that did not. And they found no statistically significant improvement on several metrics of physical well being (cholesterol checks, blood pressure checks, etc.). They did find statistically significant improvements in terms of mental health (principally depression) and financial health (apparently from not having catastrophic health expenditures). In general, physical metrics moved in the expected direction (lower blood pressure) just not sufficiently in that direction to be indistinguishable from zero. This could either be true evidence of a null relationship between insurance and health outcomes, or it could be a sign that the study was too small to capture changes in those outcomes. If you look at the study, the fact that something like 22 out of 25 metrics move in the expected (healthier) direction, even if they don’t move far in that way, suggests to me that Medicaid does improve health outcomes. But that’s a separate issue.

Two conservative, smart writers are Ross Douthat of the New York Times and Megan McArdle of the Atlantic. Both are forced to acknowledge that the Oregon Medicaid Study shows Medicaid coverage generates strong financial and mental health benefits for Medicaid recipients, but argue rhetorically: Wasn’t this about saving lives? Douthat asks, “The health care law was sold, in part, with the promise (made by judicious wonks as well as overreaching politicians) that it would save tens of thousands of American lives each year.” McArdle, drawing on the same rhetorical playbook stresses, “[W]e heard that 150,000 uninsured people had died between 2000 and 2006.” See, classic liberal over-promising and under-delivering. You told us poor people would live, not that they would be less depressed and more financially secure.

The important thing is that the Oregon Medicaid Study was a “post-treatment” survey. I’m using “treatment” in the jargon-y way. I just mean assignment via lottery to either the “treatment condition” of receive Medicaid for two years or the “control condition” of continuing insurance free for two years. It’s right there on the first page of the article: “Approximately 2 years after the lottery, we obtained data from 6387 adults who were randomly selected to be able to apply for Medicaid coverage and 5842 adults who were not selected.” To be even more precise, and requiring Douthat and McArdle to turn to the second page of the article, they collected this data via in-person interviews.

Let’s just stop right here. Dead people tell no tales. Hence they were not included in the study. The study occurred only on those people that lived to talk at the end. Medicaid could have saved 1000 lives in Oregon and this research design would not have noticed. Or Medicaid could have killed 1000 people. Same thing. This is what we like to call survivorship bias. It’s so simple, I don’t see the need to belabor the point.

But let’s imagine the study had been designed differently. At this level of power, would we have noticed? A little quick math: about 20% of Americans are uninsured, studies suggest being uninsured is associated with about 20,000 additional deaths a year nationally (U.S. population ~300m), and the control group was about 6000 people. The expected value of uninsured “excessive” deaths in this study is this rate of “excessive” deaths caused by lack of insurance per uninsured person per year times the total number in the control group. I think that gets us about 2 excessive deaths per year, or 4 excessive deaths for the period under study. I’d be very surprised if this study would be able to discern, in a statistically significant way, if Medicaid saved lives. (The death rate in the United States is 799.5/100000, meaning out of our 6000 folks, we’d expect about 96 deaths in these two years.) Even without survivorship bias.

My point: the Oregon findings in no way impugn the possibility that 20,000 Americans a year die from lack of insurance, and that Medicaid might save them. This is true solely because of survivorship bias, though sample size problems make it doubly true.

Replication: The Heart of Science

Posted on by
2

There are allegations that the Reinhart and Rogoff paper “Growth in a Time of Debt,” which has informed the current debate about debt and spending as much as any economic paper could hope, is wrong. And it’s wrong because of a data processing error, specifically what appears to be an MS Excel formula error (see here, here, or here). The original Reinhart and Rogoff working paper has over 450+ Google Scholar citations and that understates its influence in the three years since it was published.

Every dataset I have looked at has problems, and the only question is whether those problems when fixed lead the results to “break.” In practice, as a field, we seem to be okay with small fractures, things that get p-values to .07 or .11 instead of the magical .05. Journal editors don’t want to reward replication articles that merely fiddle with someone else’s hard work, though this gives authors an incentive to fiddle with their own work at the margins.

In part because replication work is rarely rewarded with journal articles, the hard work of replication, crucial for any sort of cumulative knowledge, is more or less left to graduate students in mid-level stats courses. The fact that Thomas Herndon, apparently a grad student at the UMass-Amherst Econ program, is listed as first author of this new critique makes me wonder if this sort of “replication paper” requirement is the source of this discovery.

The Smoke-Filled Room at Midwest Political Science Association

Posted on by
Reply

Four of The Smoke-Filled Room’s contributors will be presenting their papers at the Midwest Political Science Association conference starting today. We’ve pasted the paper abstracts below.

Matt Eckel: “Nationalism, Chauvinism and Inequality: Skewed Incomes, Political Elites, and the Political Economy of Xenophobia” (Panel: Thursday, April 11 12:45 pm, 21-4, Who Are We?: The Politics of Defining National Identity)

Does inequality increase the intensity of chauvinist politics? There has been substantial recent work relating socio-economic inequality to a host of political outcomes, including redistribution, partisan polarization and popular nationalist sentiment. The relationship of inequality to nationalism, in particular, has been an object of inquiry in recent years, with studies finding that unequal societies tend to have more nationalist populations. Other work on inequality and redistributive outcomes has emphasized complex dynamics through which the specific shape of income distributions shapes voter and elite incentives. In this paper I test whether there is evidence that inequality leads political elites to mobilize constituencies with more intense ethnically and culturally chauvinist appeals in order to maintain status-quo socio-economic realities. Using time series cross sectional data on inequality in OECD countries as well as measures of nationalism drawn from the Comparative Party Manifesto dataset, I find evidence that political appeals become more nationalist and chauvinist as societies become more unequal.

To download paper: http://conference.mpsanet.org/Online/Search.aspx?session=2557

Matt Scroggs: “Creating a Balance: Great Power Politics and Regional Integration“ (Panel: Thursday, April 11, 12:45-2:25, 8-3, Causes and Consequences of European Integration)

Many consider the success of the European Union to be a major blow against power-based accounts of international relations, namely realism. While there have been some attempts at applying realist theory towards European integration, namely Grieco’s “voice opportunity thesis” and Rosato’s balance of power argument, this paper will challenge the logics of both these works, as well as the liberal case put forth by Moravcsik, and will instead contrast the role of power politics and grand strategy that led to integration in Western Europe to Eastern Europe and East Asia where no such integration occurred, according to the interests of the U.S. and Soviet Union. That role, I contend, is consistent with the “realist” approach.

Natalia S. Bueno and Thad Dunning: “Race, Class, and Representation: Evidence from Brazilian Politicians” (Panel: Sunday, April 14, 8:30 am, Representation and Social Identities in Developing Countries)

 

A persistent racial gap between Brazilian citizens and their elected politicians raises the possibility of important failures of descriptive as well as substantive representation—failures that are especially puzzling in the context of Brazil’s alleged “racial democracy” as well as electoral institutions that should be favorable to racial inclusiveness. This paper uses new, original data to document for the first time the size of this representational gap. We then explore several alternative explanations for it. First, drawing on an experiment in which the race and class background of faux candidates for city council are varied at random, we find some class effects but no discernible effects of candidates’ race on voters’ support for them. Thus, the representational gap may not be readily explained by race-based voter preferences or by a failure to politicize a latent racial cleavage. Next, we explore but reject several possible institutional explanations, including discrimination by party elites and electoral rules that foster or inhibit candidate entry along racial lines. Our evidence instead suggests the importance of race-associated resource disparities that are also strongly related to electoral victory. While the mechanism through which personal assets may shape electoral outcomes should be further explored in future research, our evidence suggests the enduring importance of resource inequalities in explaining failures of descriptive representation.

Nikolay Marinov and William G. Nomikos: ”Electoral Proximity and Security Policy” (Panel: Sunday, April 14, 8:30 AM, 17-14, Democracies and International Security)

How do approaching elections a ffect the security policy states conduct? While international relations has paid some attention to this issue, existing theoretical work is scattered among many disparate arguments and the evidence does not allow researchers to identify causal relationships. We improve on both points. We identify the problem faced by democratic policy-makers near elections as a time-inconsistency problem. The time-inconsistency problem arises when the costs and benefi ts of policy are not realized at the same time, giving rise to electoral business cycles in security policy. We apply the argument to the case of allied troop contributions to Operation Enduring Freedom (“OEF”) and the International Security Assistance Force (“ISAF”) mission in Afghanistan. The exogenous timing of elections allows us to identify the causal eff ect of approaching elections on troop levels. Our fi nding of signi ficantly lower troop contributions, in the order of approximately 10 percent, near elections, is the first arguably identif ed e ffect of electoral proximity on security policy. We discuss the role of election-related incentives in eliciting suboptimal security behavior from democratic policy-makers.

The Risk of War with North Korea is Teensy

Posted on by
4

The risk of war with North Korea is small, mostly because war is a very rare event in the international system. Bennett and Stam found that the risk of war in a single directed-dyad year (e.g., U.S.-North Korea in 2013) is 0.000065. Now, this current situation is much more dangerous than your average directed dyad (e.g., U.S.-Uruguay in 1996), but my guess is even if you plugged all the variables into your handy-dandy war predicting machine, you would not get much above a 2 percent risk of war onset. With that said, since the potential costs of a North Korean conflagration likely reach hundreds of thousands of casualties, the expected value of war with the Norks is unpleasantly high (let’s say, 200,000 casualties x .02 = 4,000). By comparison, there is an approximately 100% chance that 30,000 Americans will die in car accidents this year. With all of that said, I fully support anyone that wants to engage in Doomsday Prepping, because it just makes for quality television.

Multiplicity (not starring Michael Keaton)

Posted on by
Reply

Duck of Minerva did a public service by hosting a debate between Matt Kroenig, Todd Sechser and Matthew Fuhrmann (see here, here, and here). The topic was nuclear superiority and crisis bargaining. Those are abstract words, but they are relevant for how we think about the United States’ ability to compel, say, North Korea in a crisis. But I think the debate should be of interest to anyone interested in applied methods in IR.

For me, Sechser and Fuhrmann’s arguments are the more compelling. In particular, they critique how Kroenig squeezes the appearance of more data out of a very small number of events:

Kroenig confronts a basic challenge in his empirical analysis: nuclear crises are rare.  Specifically, he has only 20 nuclear crises in his dataset (drawn from the ICB dataset). Yet he winds up with 52 observations, enough to generate a statistically significant correlation.  How does he obtain such a large dataset from such a small set of crises? The answer is that Kroenig simply duplicates each observation in the dataset, so as to double its size. A single observation for the Cuban Missile Crisis, for example, now becomes two independent events in his dataset: a victory for the United States, and a defeat for the Soviet Union.  This is inappropriate: the two observations are measuring the same event. Kroenig is not actually observing more data here; he is simply reporting the same event twice.  This is equivalent to an exit poll that lists each respondent twice in the sample – once voting for candidate X, and once voting against candidate Y – and then claims to have twice the sample size.

As quantitative methods have pushed into new areas, including areas with very few observations, their employers have suggested greater confidence than is deserved about their findings. In fact, at some point, my guess is the whole enterprise of treating dyad-years as meaningfully independent observations will come collapsing around our heads. It’s been a while since I looked at it, but Erickson, Pinto and Rader have a paper that concludes “typical statistical tests for significance are severely overconfident in dyadic data.”

Political science’s great challenge is knowing what we know. Quantitative methods are not a panacea for this problem.

Affirmative action in Brazil: the challenges of racial classification

Posted on by
3

It’s old news that Brazil is enacting social quotas – both socioeconomic and racial – for public higher education. In my earlier post, I detailed the impact this sort of policy could have on the quality of higher education.  However, before I had the chance to write a follow-up to that post, a new piece of legislation began being drafted to introduce affirmative action to the civil service.

This is not the first policy of its kind in Brazil. Yet, it is too soon to discuss the implications and effects of this law. Regardless of the final shape the bill takes, any affirmative action will have to grapple with the basic issue of identification of the beneficiaries.

In Brazil, racial classification has always been a contentious topic. For many decades, the government refused to even collect racial information, arguing that race was not a salient issue on this side of the Americas.  However, even if one agrees that there is racial discrimination in Brazil, and that part of the country’s huge inequality hinges upon race and not only class and education, the issue of racial classification is not something to be quickly dismissed. A recent New York Times  forum, for instance, shows very different perspectives.

On the one hand, Peter Fry, a leading anthropologist, argues: “[…], unlike the U.S., the majority of Brazilians do not classify themselves neatly into blacks and whites. In Brazil, therefore, eligibility for racial quotas is always a problem.”

On the other hand, Antonio Sergio Guimarães, a leading sociologist fights back:

Perhaps the biggest challenge in Brazil is the temptation to introduce a systematic verification of self-declared color or race to prevent fraud in affirmative action programs. Race and color are social constructs. It is impossible to define their borders scientifically. Passing is something inherent to this kind of classification. It can be motivated by selfish economic protection or by political altruistic reasons. The fear of fraud must be restrained to give a chance to these programs to flourish.

Ultimately, these scholars seem to be discussing an empirical and methodological issue of racial classification with wide implications for redistribution. Despite the known complexities of racial classification, much analysis relies on a single self-classification based on fixed, mutually exclusive, choices.

Bailey, Loveman and Muniz (2012) present an interesting analysis of Brazil’s racial make-up and racial inequality, taking different racial classification schemes into consideration:

They demonstrate that very different pictures of Brazil’s racial make-up are created depending on which scheme is followed. Comparing the most extreme cases, Brazil could be either 70.4% or 40.7% White. Beneficiaries of affirmative action could either comprise 29.6% or 59.3% of the population. These are hugely different percentages.

Furthermore, these different measurements are not necessarily robust.  Even if more than one measure is used, there is still a lot of incongruence.

In their paper, they go on to convincingly show that different measures also imply different mappings of income inequality between those groups. Their findings do not necessarily challenge the finding that Blacks are, on average, worse off than Whites, but they do bring more precise, rigorous, and contextual evidence to support that claim. In any case, these findings do not mean that race should be disregarded and that it does not influence social interactions in Brazil. They argue that these different measures provide more evidence that race is a multi-dimensional social construct and should be analyzed as such – there is no “true race” to be measured.

But, what do these findings tell us when discussing redistributive policies based on race? Do these inconsistencies hinder any systematic implementation of affirmative action? Or are inconsistencies (and, to some extent, fraud) a “lesser-evil”, with affirmative action a good idea despite these issues? The recent policies seem to have embraced affirmative action despite these problematic measurement issues. The consequences of these choices are still to be fully understood.

The Return of Israeli Moderation?

Posted on by
1

Not too long after the Israel-Hezbollah war, George Packer wrote an excellent profile of Israeli author David Grossman for the New Yorker. Grossman is an Israeli author who, along with several of his liberal cohort, has been engaged in a full-front assault on Israel’s hawkish foreign policy. Packer describes, in detail, how Grossman’s political opinions have evolved, like that of many Israelis, over the past few decades:

[At the time of the Yom Kippur War], his political views were conventional: Israel, surrounded by enemies, was destined to fight an eternal war, and the only imperative was survival. In 1967, the year of his bar mitzvah, Israel won the Six-Day War and occupied Gaza, the West Bank, the Sinai Peninsula, and the Golan Heights. In “The Yellow Wind,” Grossman wrote of his generation, “The surging energy of our adolescent hormones was coupled with the intoxication gripping the entire country; the conquest, the confident penetration of the enemy’s land, his complete surrender, breaking the taboo of the border, imperiously striding through the narrow streets of cities until now forbidden.” At the beginning of the occupation, Jewish families used to drive through the West Bank and Gaza on weekends, on tours organized by transportation companies like the one where his father worked; they would buy Arab kaffiyehs for next to nothing and wear them triumphantly in the streets of Hebron and Jericho. The Palestinians were crushed, and the Israelis were seduced by what Grossman calls “the temptation of strength, the temptation of arbitrariness.” At thirteen, he felt unambivalent pleasure about Israeli power. As he grew older, though, he became troubled by it; when friends or Army comrades urged him to join an outing to the occupied territories, he refused, saying, “They hate us, they don’t want us there. I cannot be like a thorn in the flesh of someone else.”

Much time have passed since this profile and since Grossman began his campaign. For years, it seemed, to those of us on the outside, that such pleas for moderation fell on deaf ears. While the settlements issue is not resolved, it appears that the Israeli “consensus” on a hardline against Iran is far from unassailable. Israel’s policy is already shifting away from military action. In a recent editorial in the New York Times, Graham Allison and Shai Feldman argue that the change of policy comes as the result of internal divisions within Benjamin Netanyahu’s government, primarily between Netanyahu and Defense Minister Ehud Barak. Indeed, several prominent Israeli political figures, including President Shimon Peres, have spoken out against unilateral military action. Moreover, as Allison and Feldman point out, the Israeli military establishment has unified in its opposition to military strikes.

Several obstacles remain, however. Most pressing, perhaps, is the possibility of a re-emboldened Netanyahu emerging from the January elections. Possible permutations of center-left coalitions consistently poll lower than Netanyahu’s coalition. In the last elections, in 2009, Netanyahu was able to form a rightist coalition despite receiving the second-most seats in the Knesset, the Israeli legislature. The centrist Kadima party, which received the most votes in 2009, was unable to form a governing coalition. It is unclear whether they will be able to unify various other centrist parties in order to succeed at this task in January. Much hope rests with Ehud Olmert, the former embattled Kadima Prime Minister. However, as Judy Rudoren argues in a Times op-ed, he faces many complicated challenges–some political, some legal, some moral–in his attempt to become prime minister once again. The titular question then can only be answered by a cautiously optimistic “maybe.”

No matter the outcome, these developments emphasize the non-unitary nature of Israeli domestic politics and foreign policy. In many ways, this mirrors a critical analytical hurdle that the field of International Relations faced several decades ago. As a recent “state-of-the-field” review article in the Annual Review of Political Science by Bruce Bueno de Mesquita and Alastair Smith argues, IR has more or less overcome this crutch. Scholars have made countless important contributions to our understanding of international politics by exploring domestic political developments explicitly.

Perhaps nowhere is this domestic turn in IR more clear than in John Mearsheimer and Stephen Walt’s controversial work on the US Israel lobby. Moreover, this analysis more or less reflects the public view of US foreign policy-making, whether true or not. It is not clear why this understanding has not extended to Israeli politics, which continues to be black- boxed in public discourse. Whatever the result of the next few months’ debate and politicking in Israel, the critical lesson for the rest of us should be not to essentialize Israeli foreign policy positions based upon the hard line it has taken so far.

For more of his thoughts on developments in Israel, follow William on Twitter.