COMPASSS network calls on journal editors and reviewers to ignore critical QCA publications

This statement openly calls on journal editors to ignore certain reviewer recommendations, and indirectly also on reviewers to ignore some of my key publications as well as publications I have co-authored with Michael Baumgartner. As we are convinced that urging journal editors to ignore some recommendations of reviewers, and indirectly reviewers to ignore our publications is not in the best interest of academic freedom, knowledge accumulation, and scientific advance, Michael Baumgartner and I have asked COMPASSS on 22 August to now also post our reply to this statement on the website of COMPASSS in order to ensure that the network remains, as it itself claims, “a place of dialogue and fruitful confrontation”. You can read our reply below:

Reply to

COMPASSS Statement on Rejecting Article Submissions because of QCA Solution Type

Michael Baumgartner (Full Professor of Philosophy, University of Bergen, Norway;
COMPASSS Advisory Board Member since 2015)

Alrik Thiem (Swiss National Science Foundation Professor, University of Lucerne;
former COMPASSS Management Board Member, 2012-2016)

On 16 August 2017, Claude Rubinson posted a public statement on the main website of COMPASSS, www.compasss.org, titled “COMPASSS Statement on Rejecting Article Submissions because of QCA Solution Type” (hereafter “the Statement”).

In the Statement, the COMPASSS Management Team and the COMPASSS Steering Committee express their concern “about the practice of some anonymous reviewers to reject manuscripts during peer review for the sole, or primary, reason that the given study chooses one solution type over another” because “the field is currently witnessing an ongoing and welcome methodological debate about the correctness of different solution types (conservative/complex, intermediate, parsimonious) when applying the methods of Qualitative Comparative Analysis (QCA) and Coincidence Analysis (CNA) to empirical data.”

The “methodological debate” mentioned in the Statement is not further specified. To our understanding and the best of our knowledge of the field, five works are implied:

(1) Baumgartner, Michael. 2015. “Parsimony and Causality.” Quality & Quantity 49(2):839-56. http://dx.doi.org/10.1007/s11135-014-0026-7.

(2) Baumgartner, Michael, and Alrik Thiem. 2017. “Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis.” Sociological Methods & Research. http://dx.doi.org/10.1177/0049124117701487.

(3) Thiem, Alrik. 2016. “Conducting Configurational Comparative Research with Qualitative Comparative Analysis: A Hands-On Tutorial for Applied Evaluation Scholars and Practitioners.” American Journal of Evaluation. http://dx.doi.org/10.1177/1098214016673902.

(4) Thiem, Alrik. 2016. QCApro: Professional Functionality for Performing and Evaluating Qualitative Comparative Analysis. R Package Version 1.1-1. URL: http://www.alrik-thiem.net/software/.

(5) Thiem, Alrik. 2017. Beyond the Facts: Limited Empirical Diversity and the Incorrectness of Qualitative Comparative Analysis (QCA). Paper presented at the 7th Annual General Conference of the European Political Science Association, Milan, Italy, 22-24 June. URL: https://www.researchgate.net/profile/Alrik_Thiem.

Both of us voluntarily declare that we have never immediately rejected an empirical manuscript, neither solely nor primarily, because of its choice of a particular solution type as we firmly believe that the foremost task of reviewers is to help guide promising manuscripts towards final publication. Thus, we have to assume that the Statement was triggered by other reviewers doing so. These reviewers must have found our arguments for the unsuitability of intermediate and conservative solutions presented in works (1) to (5) convincing enough to recommended rejection based on their own assessment of the overall scientific quality of the concerned manuscripts. We endorse the prerogative of journal editors and reviewers to favor rejection if they come to the conclusion that a manuscript does not merit publication because of its choice of an unsuitable solution type.

Moreover, we would like to emphasize that articles (1) to (3) were themselves reviewed by a larger group of anonymous experts, and found to merit publication because of the persuasiveness of their respective argument against complex/conservative and intermediate solutions. That these reviewers have recommended publication of works (1) to (3), and that other reviewers have recommended rejection of empirical manuscripts on the basis of the arguments presented in articles (1) to (3), manifestly demonstrates that there currently exists no “scientific consensus that all solutions are empirically valid, that including the full range of solutions is typically best” as the Statement claims.

We are glad to see that our arguments make an impact by persuading more and more researchers. At the same time, we expressly invite the COMPASSS Management Board and Steering Committee to enter into a formal debate with us about the validity of certain solution types for empirical research, a debate which is not currently taking place, contrary to what the Statement suggests.

We believe that the next International QCA Expert Workshop, which we both used to organize and fund in the past, would be an ideal opportunity for such a debate. Members of the COMPASSS Management Board and Steering Committee could publicly present formal arguments showing one or more of the conclusions of works (1) to (5) to be misguided—arguments which could then also be used by reviewers and journal editors to assess the methodological quality of submitted manuscripts. We are convinced that urging journal editors to ignore some recommendations of reviewers, and indirectly reviewers to ignore our publications is not in the best interest of academic freedom, knowledge accumulation, and scientific advance.

Alrik Thiem
Michael Baumgartner

[added 2/9/2017: in the meantime, other researchers have reacted to COMPASSS' call. It seems that with many researchers the network's attempt to limit the freedom to express one’s opinion in the review process, and the freedom of journal editors to follow the recommendations of their reviewers, is not going down well.

For example, Dimiter Toshkov writes in his blog: "...if the method that is used is not appropriate for the research goal and does not support the inferences advanced in the manuscript, then rejection is warranted and no further justification is needed" and "the point about non-parsimonious solutions deriving faulty causal inferences seems settled, at least until there is a published response that rebukes it." In our letter to COMPASSS, we have invited the signers of the statement to provide such a response at the 5th QCA Expert Workshop. I'm excited to see what the reaction will be. So far, they not not even published our reply on their website.]

[added 3/9/2017: The COMPASSS Management Board and Steering Committee have declined to publish our reply. In addition, they do not agree with Michael Baumgartner's complaint that the Management Board and the Steering Committee publish a statement in the name of the whole COMPASSS network, on whose Advisory Board Michael Baumgartner is. Weirdly enough, Michael Baumgartner has now been made a supporter of a statement that calls on journal editors and reviewers to ignore his own publications. Last, but not least, COMPASSS did not respond to our invitation to present a formal counterargument against our arguments presented in works (1) to (5) above that conservative and intermediate solutions fail basic methodological tests of correctness for procedures of causal inference.]