WikiLeaks and Small-n Research

I couldn’t get access to WikiLeaks this afternoon. “The site could be temporarily unavailable or too busy.” Unsurprisingly, the announcement of the publication of classified State Department documents has triggered an enormous media resonance, inviting many people to check ‘what’s up(loaded)’. From a professional perspective, I wonder whether WikiLeaks has the potential of becoming an important data source for political scientists providing them with background information about domestic and foreign politics around the globe. Of course, we can ask whether the diplomatic reports reveal more information about the world or about the authors’ perceptions of the world. The data, is subjective, but that is the case for most information that we collect in expert interviews. The good news is that this whistle-blower data is primary data. But as always, before relying on one source in small-n research (assuming that one does not mind using such sources for ethical reasons), such data should be triangulated with other sources.

Talking about triangulation, in a recent – so far unpublished – paper, Susumu Shikano, Stefanie Walter and I analyzed different data aggregation strategies commonly used in small-n research. In the paper, we are interested in how best to aggregate different sources under different conditions in triangulation. Using computer simulations we so far tested a set of five very simple aggregation strategies, namely, a random selection of sources, a simple average, a weighted average, the mode and a winner-takes-it all strategy.

In our simulations we a-priori define a uni-dimensional continuous scale for a concept whose true value is assumed to be 50 without loss of generality. We further assume that every expert is uncertain about this true value to some degree. That is, every expert recognizes the true value with certain cognition error. For the errors, we assume a normal distribution whose expected value equals zero. That is, the probability of small errors is higher than that of large errors. We differentiate this probability of errors between experts with different information levels (technically, this is reached by varying the standard deviation).

Once we define the number of experts holding different information levels, we randomly draw information from the corresponding normal distributions independently for each expert. We repeat this random draw and aggregation a 1000 times and thereby obtain 1000 measures for each aggregation rule. These results are, in turn, evaluated in terms of the ‘true’ value. For this purpose, we utilize the mean absolute error (MAE).

The left panel of the figure below (please follow the link) highlights our simulation results with one well-informed source (stdev of 2) and an increasing number of worse informed sources (stdev of 20). In the right panel we increase the number of well-informed sources.

When assuming that small-n researchers commonly only use up to six sources we find that a weighted average is to be recommended. Assuming unsystematic errors, the simple average in our simulations only reach a similar performance when using six or more sources. The winner-takes-it-all strategy displays a satisfactory performance. The winner-takes-it-all and the weighted average, however, demand a qualification of the sources. But how would you, for instance, classify the documents released by WikiLeaks?


Tolstoy and Rhetorical Action

Following up on Fabrizio Gilardi’s recent excursions into world literature and in commemoration of Leo Tolstoy who passed away 100 years back on November 20th 1910, I would like to share a literary illustration of the social influence concept of shaming (i.e. the public expose of inconsistencies and illegitimate behavior). I cite from Tolstoy’s Kreutzer Sonnata in which during a train ride, the first-person narrator clashes with an old merchant about gender equality. The merchant: “Ja, so ist’s wenn man versäumt, den Frauen rechtzeitig die Kandare anzulegen, dann ist’s gefehlt.” “Aber Sie haben doch eben selbst erzählt, wie sich verheiratete Männer auf der Messe in Kunawino belustigen”, sagte ich; ich konnte diese Bemerkung nicht unterdrücken. “Das ist ein Ding für sich”, sagte der Kaufmann und brütete schweigend vor sich hin.

Barrosobama – Add on…

Laura Gies just send me these nice little wordclouds that so neatly illustrate my blog from yesterday. The first one is a cloud capturing the essence of Obama’s January 2010 State of the Union Address.

This one relates to Barroso’s speech at the European Parliament last week.

Of course, I take them as confirmations of yesterday’s blog 😉

Barrosobama – Same States of Different Unions?

Last Tuesday, we witnessed the first so-called “state of the Union” address by a European Commission President. Drawing on the American example – following article II, sections 3 of the Constitution to the United States of America the US President annually informs the houses of Congress about the ‘State of the Union, and recommend to their Consideration such Measures as he shall judge necessary and expedient’ – José Manuel Barroso held a speech, described as ‘upbeat’ and ‘presidential-type’ ( Despite the question whether this is really something new – Commission Presidents already in the past had presented their work programs to the European Parliament on a regular basis – and whether Barroso is really the head of the EU’s executive – remember van Rompuy – and whether it was a good idea of threatening the MEPs with a fine in case of their abstention, I was wondering how strongly Barroso’s speech mirrored Barack Obama’s state of the Union address delivered on January 27th 2010.

The two speeches differ in length: Obama uses 7400 words as compared to Barroso’s 4400. In terms of content, both speeches center on the state of the economy, (‘smart’) growth, the creation of jobs (preferentially in the eco-industries), innovation, and energy questions. While Barroso promotes his 2020 reform agenda and a further deepening of the single market, Obama defends his health care program. Both speakers share concerns about civil rights. Obama focuses stronger on security and geostrategic issues. Barroso expresses his wish of seeing Europe become a global player, a ‘global leader’ but for the sake of promoting climate change, international cooperation and the promotion of the EU’s values, as for instance the respect of human rights. Both speakers underline their interest in the Doha round.

The two speeches address different audiences and they differ in terms of style. Commission President Barroso does not greet his ‘fellow Europeans’ and he delivers the basic structure of a work program – without actively discussing its pros and cons and illustrating and supporting his claims with examples from l’Europe profonde. But there are some attempts not to sound too bureaucratic: ‘This is Europe’s moment of truth. Europe must show that it is more than 27 different national solutions. We either swim together or sink separately. We will only succeed if, whether acting nationally, regionally or locally we think European.’ Compared to Barroso, Obama celebrates a mass, interacting and flirting with the audience: ‘I thought I’d get some applause on that one’. Or his comment on the bank bailout: ‘I hated it. You hated it. It was about as popular as a root canal.’ Obama cheers up his audience – ‘Let’s get it done. Let’s get it done’ – in a blood, sweat and tears fashion basing his optimism on the American spirit of ‘great decency and great strength’. And yes, Barroso doesn’t end with: ‘And god bless the European Union’.

But he hasn’t got anything on!

In Hans Christian Andersen’s short tale ‘The Emperor’s new Clothes’, it is a child pointing out the obvious fact: “But he hasn’t got anything on.” Simple truths often evade more sophisticated observers. And similarly, in academia, we often do not bother to ask and (for that matter) answer seemingly obvious questions.

Recently, in a meeting reuniting ten experts of EU decision-making, a colleague asked what actually drives the European Commission to submit its proposals either in the form of directives or regulations. While we all had an idea about the legal differences between these two provisions – regulations are binding in entirety and directly applicable in all Member States, directives leave to national authorities the choice of form and methods of transposition (Art. 288 of the Treaty on the Functioning of the European Union; former Art. 249 EC Treaty) – nobody of us, including the author, was certain about when the Commission opted for directives and when it pulled the regulations card. The legal answer is that in “most instances the Treaty leaves open the choice whether to proceed by way of regulation, directive or decision” (Craig & De Búrca 2008: 83). Fair enough. But the answer highlights that there is great room for strategic action – usually a nice playing ground for political scientists.

Commission submission p.a. (source: EULO)

The graph based on our new EULO dataset (more information on this is to follow) plots the yearly amount of Commission proposals in the form of regulations, directives and decisions (decisions are binding but they are not addressed to all member states).

Continue reading

Talking Politics: Why not set up a ‘Political Science Live’ archive?

American political science in Europe today is mostly equated with quantitative social science. See quantification as good or bad, it’s just one side of the coin. American political science has more to offer. Check out, for instance, the marvelous “Conversations with History”. In this collection of unedited audiovisual interviews Berkeley’s Harry Kreisler talks to important academics and activists and makes them reflect on their lives and their work. ‘Political awakenings’ are discussed but the conversations also touch upon current political, economic or cultural questions. You can, for instance, get a firsthand report of Ernst Haas’ youth in Nazi-Germany, hear Barry Eichengreen comment the global economic crisis or learn why Stephen D. Krasner considered his academic background as useful for his former work as director of policy planning at the state department. The basic idea behind the platform is to “capture and preserve through conversations and technology the intellectual ferment of our times.”

Continue reading