A compendium of methods and stats resources for (social) psychologists
This page helps me recover papers or websites that I use regularly when planning or analyzing research studies. I hope you will find it useful as well.
Olivier Klein.
List of the topics covered in this page:
Methodological Issues with Online Surveys
Power estimation and effect sizes
Social networks (sharing resources & knowledge)
- Bill McGuire’s important annual review chapter about creative hypothesis generation in psychology.
- Paper by Pirlott & McKinnon on how to design studies with mediation in mind
- The classic paper by Spencer, Zanna & Fong on why designing experiments is preferable to mediational analysis in examining causal processes.
- Document destined to Ph.D. students at the University of Guelph explaining how to avoid questionable research practices in planning and reporting experiments.
Methodological issues with online surveys
- Paper by Zhou & Fishbach showing validity problem with attrition in online surveys and how to deal with them.
- Identifying careless responses in survey data. Paper by Meade et al.
- Seriousness checks to improve the reliability of online surveys. Paper by Aust et al.
- Detecting and deterring insufficient efforts in responding to surveys. Paper by Huang et al.
- Randomly assigning people to different conditions in limesurvey.
- Clark & Watson’s (1995) paper on scale development.
- Chapter by Malte Elson on question wording and item formulation
- Excellent website with guidelines on designing rating scales.
Power estimation & Effect sizes
- Online calculator for power estimation in mixed models
- Clear introduction to estimating and reporting effect size by Daniël Lakens (Frontiers). Check Katherine Wood’ online version (shiny): it is even more user-friendly.
- G*Power, the free software that calculates power for a variety of designs. The manual is here.
- Pangea: A web applet that computes power for General Anova designs. By Jake Westfall. Thanks, Almudena Claassen, for pointing me to this!
- Computing power for interactions involving one continuous and one dichotomous variable. Online application designed by myself.
- When there is more than one within subject factors, G*Power can’t compute power. The best solution is to run simulations but that requires programming skills. D’Amico et al. proposed a method using SPSS’s MANOVA procedure. Here is also another paper using this method for regression, correlation and simple anova designs.
- Here is how to calculate power for a 3-way ANOVA in G*Power
- Converting effect size (e.g., from d to eta square, etc). Excel page here.
- Sequential data analysis. Great method for maximizing power and minimizing sample size at the same time. EJSP paper by Daniël Lakens.
- How many participants do I need to test a moderation of the effect I found in my first study? Many. Post on Data Colada explaining this.
- A tutorial by myself on Determining sample size in social psychology (with tables). French.
- Sample size planning adjusting for publication bias and uncertainty by Anderson et al.
- Using thematic analysis in psychology. Excellent introduction by Braun & Clarke to a fully qualitative method for analyzing texts.
- Three approaches to qualitative Content Analysis. Paper by Hsieh & Shannon.
- And here are two resources on discourse analysis recommended by Theofilos Gkinopoulos:
- Discourse Analaysis: The identification of interpretive repertoire by Margaret Wetherell & Jonathan Potter.
- Reynolds, J. & Wetherell, M. (2003). The discursive climate of singleness: The consequences for women’s negotiation of a single identity. Feminism & Psychology, 13(4), 489-510 (a helpful and clear study of the use of interpretative repertoires, recommended also in Wiggins’s book on theory & methods in discursive psychology as an example article).
- Open Science Framework: Ideal place to store all materials relevant to a research project.
- Aspredicted.org: Website for easily preregistering studies.
- Guidelines + preregistration template by Van ‘t Veer and Giner-Sorolla, JESP, 2016.
- Here is a very simple and well done preregistration worksheet by Elizabeth Dunn.
- Another template at the OSF here.
- Using graphs instead of tables in political science: Great paper by Kastellec & Leoni showing how to replace tables by graphs. Applies to social psychology as well
- Our paper “Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median”
- Bakker & Witchers’s paper arguing against outlier removal.
Statistical Inference (Fisher/Neymann-Pearson)
- Jacob Cohen’s paper, Things I’ve learned so far. Essential reading that covers many of the core issues psychologists should be attuned to when conducting (inferential) statistical analyses.
- Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Paper by Greenland et al. See also American Statistical Association’s statement on p values.
- One simple effect is significant, the other not but no interaction. Paper by Gelman on this.
- Is it a problem to use parametric stats on likert scales when the sample size is low or the distribution far from normal? Usually not according to this paper by Geoff Norman.
- Using covariates when testing for interactions. Paper by Yzerbyt et al.
- p curve
- Testing that the null is true without Bayes. Blog post by Daniël Lakens on Equivalence testing and the paper now in SPPS.
- A post by Heino Matti on false expectations about the relation between p values and sample size. Includes great vizualisations.
- Aligning scientific reasoning and statistical inference: Short “Science” paper by Steven Goodman on misunderstandings in statistical inference and their impact on scientific progress.
- Great paper by Miller and Chapman on misunderstandings surrounding the interpretation of ANCOVA.
- Ever wondered about the meaning of degrees of freedom? Check out this excellent tutorial by Ron Dotsch(thanks, Rui Costa Lopes for the suggestion!) .
- Partial Least Square Regression: I am not very knowledgeable on this one but colleagues recommended that I include this method for testing causal models on this page. So here are two links: one recommended by Davide Del Cason and the other recommended by Carole Fantini.
- Mixed Models: Introduction to treating stimuli as random factors and code for common statistical software by Westfall et al.
- Follow-up on mixed models: Annual review chapter by the same authors addressing various research designs
- Significance testing in lme4
- Should you fit the “maximal model”? Parsimony in model construction. Paper by Bates et al.
- Centering predictors in mixed models. Paper by Enders & Tofighi.
- David Kenny’s simple and excellent mediation page.
- Broader overview of mediation and moderation. By Judd et al (2014).
- Interactions do not tell us when but also tell us how. Nice paper by Jaccoby & Sassenberg (2011).
- A paper by Rik Pieters explaining when mediation analysis is warranted.
- Paper by Pirlott & McKinnon on how to design studies with mediation in mind
- The classic paper by Spencer, Zanna & Fong on why designing experiments is preferable to mediational analysis in examining causal processes.
- Zoltan Dienes’ very useful webpage on Bayesian stats for beginners (including online calculators).
- How to get the most of nonsignificant results? Paper by Zoltan Dienes based on a a Bayesian approach.
- Is there a free lunch in inference? Forceful advocacy of the Bayesian approach by Rouder et al. Very clear for nonspecialists.
- Short intro to Bayesian stats with R examples by Fabian Dablander.
- Paper by Goodboy & Kline: Statistical and practical concerns with research featuring structural equation modeling. Good primer on some of the errors you want to avoid!
- Psychological Methods discussion group (Facebook)
- PsychMAP (Facebook)
- Annotated SPSS Output for Logistic Regression
JASP: a free statistical software that also performs bayesian tests.
Jamovi: Even better. Open source.
- If you want to learn R by yourself check these resources designed by Sean Murphy.
- The perfect t-test. R program that reports the results of a t-test completely formatted, with graphs, tests of assumptions, etc. By Daniël Lakens.
- Bodo Winter’s tutorials on mixed models.
- Tutorial on logistic regression.
- Swirl: Package for learning R from inside R.
- Guide de démarrage pour GGPlot. French.
- Making it pretty: Plotting 2-way interactions with GGplot2. Nice tutorial & code.
- Implementing Edward Tufte’s recommendations for cool looking graphs using R.
- Appropriate categorical variables coding schemes for linear regression in R.
- APA’s guidelines on Responsible Conduct of Research
- Using deception ethically: Useful guidelines by Antonio Pascual Leone et al.
- Shiny Web Apps for designing experiments and analysing data
- Rpsychologist: All kinds of vizualisations of common statistical procedures. Splendid for pedagogical purposes especially.
- Nice post by Chris Holdgraf on designing and interpreting funnel plots.
- Paper by Butts et al. on the source of common errors in the interpretation of cutoff criteria for widely used stats (including .70 for Cronbach’s alpha)
- Blog post by Dick Morey on the dangers of averaging data.
- On a similar topic, paper by Colin Leach on how to integrate the person with macro-level processes (and look differently at regression plots).
- Paper by LeBel & al summarizing criticisms of open science and advocacy of high powered research movement in social psych. and addressing them. Useful to find all the important recent references.
- Statcheck: Online tool to check whether there any errors in your paper. Just upload a PDF.
