Corrigendum to “Lowering the Threshold of Statistical Significance to p < 0.005 to Encourage Enriched Theories of Politics” and “Questions and Answers: Reproducibility and a Stricter Threshold for Statistical Significance”

Although The Political Methodologist is a newsletter and blog, not a peer-reviewed publication, I still think it’s important for us to recognize and correct substantively important errors.  In this case, I’m sad to report such errors in two things I wrote for TPM. The error is the same in both cases.

In“Lowering the Threshold of Statistical Significance to p < 0.005 to Encourage Enriched Theories of Politics,” I claimed that:

When K-many statistically independent tests are performed on pre-specified hypotheses that must be jointly confirmed in order to support a theory, the chance of simultaneously rejecting them all by chance is αK where p < α is the critical condition for statistical significance in an individual test. As K increases, the α value for each individual study can fall and the overall power of the study often (though not always) increases.

This argument is offered to support the conclusion that “moving the threshold for statistical significance from α = 0.05 to α = 0.005 would benefit political science if we adapt to this reform by developing richer, more robust theories that admit multiple predictions.”

Similarly, in “Questions and Answers: Reproducibility and a Stricter Threshold for Statistical Significance,” I claimed that:

Another measure to lower  Type I error (and the one that I discuss in my article in The Political Methodologist ) is to pre-specify a larger number of different hypotheses from a theory and to jointly test these hypotheses. Because the probability of simultaneously confirming multiple disparate predictions by chance is (almost always) lower than the probability of singly confirming one of them, the size of each individual test can be larger than the overall size of the test, allowing for the possibility that the overall test is substantially more powerful at a given size.

This reasoning, which is similar to reasoning offered in Esarey and Sumner (2018b), is incorrect; it would only be true when all predicted parameters were equal to zero. When the alternative hypothesis is that multiple directional predictions for parameters,  for example βi > 0 for i 1…K, separate t-tests rejecting each individual null (βi 0) separately using t-tests with size α will jointly reject all the null hypotheses at most α proportion of the time. The key insight is that the joint null hypothesis space includes the possibility that some βi parameters match the predictions while others do not; if (for example) β1 = 0 and all other βi=/=1 are very large, the probability of falsely rejecting the joint null hypothesis is the α for the test of β1. As we note in Esarey and Sumner (2018a), this is discussed and proved in Silvapulle and Sen (2005, Section 5.3), especially in proposition 5.3.1, and in Casella and Berger 2002, Section 8.2.3 and 8.3.3. Silvapulle and Sen cite Lehmann (1952); Berger (1982); Cohen, Gatsonis and Marden (1983); and Berger (1997) (among others) as sources for this argument. Associated calculations (such as that in Figure 4 of “Lowering the Threshold of Statistical Significance to p < 0.005 to Encourage Enriched Theories of Politics”) are based on the same error and therefore incorrect.

The upshot is that my argument for making additional theoretical predictions in order to facilitate lowering the threshold for statistical significance to α = 0.005 is based on faulty reasoning and incorrect.

I plan to post this correction as an addendum to both of the print editions featuring these articles.

References

Berger, Roger L. 1982. “Multiparameter Hypothesis Testing and Acceptance Sampling.” Technometrics 24(4):295–300.

Berger,  Roger  L.  1997.  Likelihood  ratio  tests  and  intersection-union  tests.  In  Advances  in statistical decision theory and applications, ed. Subramanian Panchapakesan and Narayanaswamy Balakrishnan.  Boston:  Birkhäuser pp. 225–237.

Casella, George and Roger L.. Berger. 2002. Statistical Inference, Second Edition. Belmont,
CA: Brooks/Cole.

Cohen, Arthur, Constantine Gatsonis and John I. Marden. 1983. “Hypothesis testing  for marginal probabilities in a 2 x 2 x 2 contingency table with conditional independence.”   Journal of the American Statistical Association 78(384):920–929.

Esarey, Justin and Jane Lawrence Sumner. 2018a. “Corrigendum to Marginal Effects in Interaction Models: Determining and Controlling the False Positive Rate.” Online. URL: http://justinesarey.com/interaction-overconfidence-corrigendum.pdf.

Esarey, Justin and Jane Lawrence Sumner. 2018b. “Marginal Effects in Interaction Mod-  els: Determining and Controlling the False Positive Rate.” Comparative Political Studies 51(9):1144–1176. DOI: https://doi.org/10.1177/0010414017730080.

Lehmann, Erich L. 1952. “Testing multiparameter hypotheses.” The Annals of Mathematical Statistics pp. 541–552.

Silvapulle, Mervyn J. and Pranab K. Sen. 2005. Constrained Statistical Inference: Inequality, Order, and Shape Restrictions. Hoboken, NJ: Wiley.

Posted in Uncategorized | 2 Comments

Papers Written by Women Authors Are Cited Less Frequently, but the Etiology of this Finding is Complex

Justin Esarey, Wake Forest University
Kristin Bryant, Rice University

Synopsis

A recent symposium in Political Analysis, anchored around Dion, Sumner and Mitchell (2018), discusses their finding that articles authored by women are more likely to cite at least one paper authored by women. Our contribution to this symposium (Esarey and Bryant, 2018) noted that articles in the Dion, Sumner, and Mitchell (2018) data set with at least one female author are cited no more or less often than male-authored articles once we control for the publishing journal and the number of authors. In this paper, we present additional findings that place the results of our original paper into a broader context. This context is important to fully understand how scholarship by women is utilized by the discipline, how scholars’ careers are impacted as a result of this utilization, and how we might achieve greater gender parity in the field.

When looking at the the unadjusted data set, articles with at least one woman author are in fact cited fewer times on average. It is plausible that this citation gap does represent a substantively meaningful barrier to the advancement of women in the discipline. As we reported in Political Analysis, papers with women authors are no more or less likely to be cited once the number of authors and the publishing journal are controlled for via linear regression. However, simply controlling for author count is insufficient to eliminate the gender disparity in citations: controlling for the publishing journal is crucial. An implication is that women may be systematically disadvantaged in the field, but that this disadvantage is not a function of discrimination against women when articles are chosen to be cited. Instead, consistent with the findings of Teele and Thelen (2017), we find that articles in the most-cited journals of the discipline are less likely to have women authors. The etiology of that relationship (and the citation gender gap that it creates among political scientists) is difficult to unravel.

Full Text


Replication File

A replication file for this paper is available at https://doi.org/10.7910/DVN/XC76G3.

Posted in Uncategorized | Leave a comment

International Methods Colloquium: 2018-2019 Schedule!

On behalf of the advisory board (Michelle Dion, Cassy Dorff, Jeff Harden, Dustin Tingley, and Chris Zorn), I am pleased to announce the schedule of International Methods Colloquium series talks for the 2018-2019 academic year!

The International Methods Colloquium (IMC) is a weekly seminar series of methodology-related talks and roundtable discussions focusing on political methodology; the series is supported by Wake Forest University and was previously supported by Rice University and a grant from the National Science Foundation. The IMC is free to attend from anywhere around the world using a PC or Mac, a broadband internet connection, and our free software. You can find out more about the IMC at our website:

http://www.methods-colloquium.com/

where you can join a talk in progress using the “Watch Now!” link. You can also watch archived talks from previous IMC seasons at this site. Registration in advance for a talk is encouraged, but not required.

Note that all talks begin at 12:00p Eastern Time and last precisely one hour.

Here is our schedule of presenters (and a link to our Google Calendar):

Fall Semester
  1. Oct 12: Matthew Blackwell, Harvard [register to attend]
  2. Oct 19: Masha Krupenkin, Stanford [register to attend]
  3. Nov 2: Roundtable on Gender, Citations, and the Methodology Community with Michelle Dion (McMaster), Sara Mitchell (Iowa), Dave Peterson (Iowa State), and Barbara Walter (UCSD) [register to attend]
  4. Nov 9: Luke Keele, University of Pennsylvania [register to attend]
  5. Nov 16: Kevin Munger, Princeton/Penn State [register to attend]
  6. Nov 30: Pablo Barbera, London School of Economics [register to attend]
Spring Semester
  1. Feb 1: Michelle Torres, Rice [register to attend]
  2. Feb 8: Marcel Neunhoeffer, Mannheim [register to attend]
  3. Feb 15: Winston Chou, Princeton [register to attend]
  4. Feb 22: Erin Rossiter, WUSTL [register to attend]
  5. Mar 1: Matthew Tyler/Christian Fong, Stanford [register to attend]
  6. Mar 8: Rob Carroll, Florida State [register to attend]
  7. March 29: Carlos Carvalho, University of Texas Statistics [register to attend]

Additional information for each talk (including a title and a link to a relevant paper) will be released closer to its date.

Please contact me if you need any more information; I hope to see many of you there!

Posted in Uncategorized | Leave a comment

Using Sequence Analysis to Understand Career Progression: An Application to the UK House of Commons

Matia Vannoni, IGIER, Bocconi University
Peter John, Department of Political Economy, King’s College London

Abstract: We argue that sequence analysis, mainly used in sociology, may be effectively deployed to investigate political careers inside legislatures. Career progression is a classic topic in political science, but political scientists have mainly examined access to legislatures. Although data reduction methods, for instance, can provide insight, we argue that sequence analysis can be used to understand better the career patterns inside parliaments. In this paper, we explain the method. Then we show how it can describe steps in political careers and map different patterns of advancement. We apply sequence analysis to a case study of MPs in the UK House of Commons from 1997 to 2015. We describe the variety of career paths and carry out regression analysis on the determinants of MP career progression.

Full Article

 

Online Appendix

 

Replication File

Replication files for this paper are located at: https://doi.org/10.7910/DVN/I8YHPT.

Posted in Uncategorized | Leave a comment

MZES Open Social Science Conference 2019: Practicing New Standards in Transparency and Reproducibility

I received this message from Alexander Wuttke of the Mannheim Centre for European Social Research and the Leibniz Institute for the Social Sciences, announcing a new conference on Transparency and Reproducibility in the Social Sciences. This conference may be of interest to readers of The Political Methodologist!

— begin announcement —

MZES Open Social Science Conference 2019: Practicing New Standards in Transparency and Reproducibility

This conference is a forum for practicing and discussing credibility, transparency and replicability in the social sciences.

About a decade ago, John Ioannidis claimed that “most published research findings are false”. While seeming outrageous at the time, a growing body of meta-science research in the behavioral and social sciences substantiated this claim, causing uncertainty about the trustworthiness of published scientific findings. We believe that threats to the validity of published findings in the social sciences are widespread and systemic. Therefore, this conference promotes introspection about the current state of social science research and exchange on the opportunities for institutional and methodological improvement in the future.

The conference is supported by the Berkeley Initiative for Transparency in the Social Sciences (BITSS) and will take place from 25-27 January 2019 in Mannheim, Germany.

Conference Website – twitter

Keynote speakers:

  • Jeremy Freese (Stanford University)
  • Thomas König (APSR, University of Mannheim)
  • Arthur Lupia (OSF, University of Michigan)
  • Julia Rohrer (100% CI, Leipzig University)

Participate in the conference:

  • Give a talk: We call for researchers to advance discussion, debate, literature synthesis, or methods in open social science. We welcome methodological advances, e.g., p-curve analysis, systematic reviews, pre-analysis planning, and replication. We welcome general research findings that apply best practices of open science while conducting the research – Abstract submission DL: 22 August 2018  Read more
  • MZES-GESIS Pre-Registration Challenge:  We call for researchers to participate in a competition to win funding or survey time for the most innovative and rigorous pre-registration plan for a social science study. – Abstract submission DL: 22 August 2018  Read more
  • OSSC19 Crowdsourced Replication Initiative: We call for researchers to replicate and expand a previously published cross-national macro-comparative study. The goal is to explore and develop crowdsourcing methods and generate research surpassing what a single researcher could achieve. The replication comes from the field of immigration and social policy, but we encourage social science researchers of all disciplines and levels to participate. All full participants will be co-authors on the final paper. – Registration DL: 27 July 2018   Read more
  • Participate as a guest in Mannheim during the conference or during the subsequent Open Science Workshop, offered in collaboration with the Berkeley Initiative for Transparency in the Social Sciences (BITSS). Or use the live stream online.

Organizing Committee

Nate Breznau (MZES, University of Mannheim)
Eike Mark Rinke (MZES, University of Mannheim)
Alexander Wuttke (MZES, University of Mannheim)

Conference website: http://www.open-socsci.org/
Twitter: @opensocsci
#ossc19

Posted in Uncategorized | Leave a comment

New (but old) Print Edition Released!

The blog format of The Political Methodologist has made the release of print editions less pressing, as new content is immediately distributed to readers through our web presence. However, we still provide a LaTeX-compiled print edition, primarily so that our content can be properly cited in other work and circulated in print as necessary. This particular issue has been compiled rather belatedly: this is the Fall 2017 edition, and the content in this issue has been available on our website for months. We apologize for the lateness of this issue; at this time, TPM labors in the absence of any editorial assistance.

Posted in Uncategorized | Leave a comment

Report on the 2018 Asian Political Methodology Meeting

[This post is contributed by Jong Hee Park, Seoul National University.]

On January 11 and 12, 2018, the fifth Asian Political Methodology Meeting was held at Seoul National University, Republic of Korea. The meeting was co-sponsored by the Department of Political Science and International Relations at Seoul National University and Program for Quantitative and Analytical Political Science of Princeton University.

This year’s program, available at https://asiapolmeth.princeton.edu/online-program, had eight sessions including two keynote speeches and two poster sessions. In total, 14 papers and 30 posters were presented. In total, 95 registered participants (25 foreign and 70 local) attended the entire conference and many unregistered participants also joined the conference. Nationalities of participants were from Australia, China, Germany, Hong Kong, Ireland, Japan, Republic of Korea, Singapore, United Kingdom, and United States.

The invited keynote speaker was Prof. Michael D. Ward from Duke University. Prof. Michael D. Ward gave a talk about how to analyze relational (network) data using statistical methods. Another keynote speaker was from KAIST, Republic of Korea: Prof. Meeyoung Cha. Prof. Meeyoung Cha presented a method of detecting fake news in online social media using machine learning techniques. After the keynote speech by Prof. Ward, the conference moved to a session of “Big Data in Political Methodology,” “Experimental Methods,” and “Bayesian Analysis.” The first day of the conference ended with the first poster session, consisting of faculty and post-doc participants.

The second day of the conference started with a theme of “Political Methodology for Lobby,” and then moved to a session of “Statistical Methods for Representation.” After the second poster session by graduate students, Prof. Cha gave a keynote speech and the conference finalized the program with a session of “Analyzing Congress using Statistical Methods.”

To make this conference successful, six graduate students at Seoul National University voluntarily contributed their time and resources for two months. Their names are Soonhong Cho, Suji Kang, Doeun Kim, Sunyoung Park, Sooahn Shin, Hyein Yang in alphabetical order. On behalf of the program committee, we sincerely appreciated their help and contribution.

 The program committee for this conference included Jong Hee Park (committee chair and local host: Seoul National University, Republic of Korea), Fang-Yi Chiou (Academia Sinica, Taiwan), Kentaro Fukumoto (Gakushuin University, Japan), Benjamin Goldsmith (University of Sydney, Australia), Kosuke Imai (Princeton University, USA), and Xun Pang (Tsinghua University, China).

The 2019 Annual Meeting will be held in Kyoto, Japan. We look forward to seeing you in Kyoto next year!

Jong Hee Park is Professor, Department of Political Science and International Relations, Seoul National University, Republic of Korea.

 

Posted in Uncategorized | Leave a comment