A compendium of methods and stats resources for (social) psychologists

A compendium of methods and stats resources for (social) psychologists

This page helps me recover papers or websites that I use regularly when planning or analyzing research studies. I hope you will find it useful as well. I am open to suggestions for things to add or remove (okleinATulb.ac.be or @olivier_klein on Twitter).

Olivier Klein

List of the topics covered in this page:

Research Design

Causality

Software for designing experiments

Online Surveys

Measurement

Sampling

Power estimation and effect sizes

Effect size

General Linear Model

Mixed Models

Other

Qualitative Methods/Text Analysis

Open science (general)

Preregistration

Data Management and Preparation

Displaying data

Testing

Inference

Linear Regression and General Linear Model (t-test, ANOVA, Multiple Regression, etc)

Mediation and Moderation

SEM

Bayesian

Mixed Models

Logistic Regression

Meta-Analysis

Statististical Software

SPSS

R

Learning R

General Reference

Markdown

Freely Available Datasets

Other

Social networks (sharing resources & knowledge)

Ethics

Other stuff

 

Research design

Causality/Endogeneity/Theory

  • Bill McGuire’s important annual review chapter about creative hypothesis generation in psychology.
  • Cronbach & Meehl’s classic paper on construct validity (the foundation of  psychological theorizing thereafter).
  • Paul Meehl’s 1990 paper on appraising theories in psychology, where he qualifies his earlier paper comparing theorizing in physics and psych. See also this more accessible 1967 paper on the uninterpretability of research summaries in psychology and his delightful 1973 chapter “Why I do not attend clinical case conferences” (that inspired Baruch Fischhoff).
  • Patrick Langford (also known as Psychbrief) made summary of Paul Meehl’s lectures (that are available online) as well as of some of his most important papers. Impressive! All of this is available here.
  • Barry Markovsky on evolution and nebulousness in social psychological theories.
  • Klaus Fiedler’s thoughts on the cycle of theory formation in (social) psychology (paywall).
  • Gerd Gigerenzer’s personal reflections on theory and psychology.
  • Yarkoni and Westfall’s radical take on the matter: we should choose prediction (using machine learning) rather than explanation in psychology.
  • General and simple introduction to the problem posed by “endogeneity” (ie., when a causal variable is correlated with the error term of the DV as is  usually the case in non-experimental designs) in testing causal relationship, and how to deal with them, by Antonakis et al. This is applied to leadership research but the points made apply to social psych as well.
  • If you only have a few minutes to get acquainted with this crucial issue, check out this very short paper by Hernan called “The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data”.
  • How to move from statistical association to causation? Paper by Genetian et al. introducing researchers in developmental psych. to the “instrumental variables” approach to deal with this issue. Again, very accessible and useful for social psychologists as well. The more extensive paper on instrumental variables in the social sciences by Kenneth Bollen is available here  (paywalled).
  • Causal Models and Learning from Data: paper by Maya Petersen & Mark Van der Laan on applying causal modeling to epidemiological data
  • Julia Rohrer’s excellent paper “Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data”. Proposes new (to social psychologists) tools to infer causation from observational data.
  • In a similar vein, Judea Pearl, who wrote the remarkable “Book of Why” and is the “father” of such graphical models, explains in this short paper how to properly assess causality using linear models using path diagrams. It remains challenging for psychologists not used to equations but is worth the effort.
  • Are randomized controlled trials the panacea for establishing causality? Deaton and Cartwright argue against it in this paper (preprint) extracted from a special issue of Social Science and Medicine that addresses this topic at length (including a response to this paper; but the rest of the issue is behind a paywall).
  • Thinking about causality in complex interventions. Paper by Rickles (2009) with paywall. See this on a similar topic.

      Many thanks to Holger Steinmetz & Djouaria Ghilani for the suggestions in this section! 
Experimental Design

  • Paper by Pirlott & McKinnon on how to design studies with mediation in mind
  • An excellent video by Marco Perugini on sound research design and power analysis.
  • The classic paper by Spencer, Zanna & Fong on why designing experiments is preferable to mediational analysis in examining causal processes.
  • Document destined to Ph.D. students at the University of Guelph explaining how to avoid questionable research practices in planning and reporting experiments.
  • Paper by Biglan et al. on time series experiments: a solution when randomized controlled trials are too costly or difficult.
  • When you can’t access large samples, randomization tests may be a solution. Check out this and this.
  • Resources for designing Vignette studies in this twitter thread. 
  • “Selecting a Within- or Between-Subject Design for Mediation: Validity, Causality, and Statistical Power”. Paper by Amanda Montoya.

Software for designing experiments

  • Psytoolkit: free to use software to conduct experiments online.
  • Psychopy: open source software for conducting psychology experiments (offline). There are two interfaces: a user friendly one and a powerful one using python programming language.
  • Psyscope: software to design experiments using Mac OS.
  • Gorilla: free user friendly software for conducting experiments online. However, this is not a free solution: the researcher pays per subject.

Sampling

  • A lucid paper by Neil Stewart et al. introducing readers to crowdsourcing platforms in cognitive science with their benefits and drawbacks (another version without paywall here).
  • Paper by Goodman et al. on strength and weaknesses of Amazon Mechanical Turk Samples (applied to consumer research).
  • This is the “classic” 2011 and much cited paper by Buhrmeister et al. suggesting that Amazon Mechanical Turk provided high quality data.
  • A list of crowdsourcing platforms provided by Gorilla Science.
  • Classic paper by Henrich et al. on the use of samples from Western, Educated, Industrialized, Rich and Democratic (WEIRD) societies in psychology.
  • An editable list of resources on online studies (thanks, Gabriele Paolacci).
  • OSF resource on sampling (for workshop @ #SIPS 2019).

Online surveys

  • Paper by Zhou & Fishbach showing validity problem with attrition in online surveys and how to deal with them. Super important!
  • Identifying careless responses in survey data. Paper by Meade et al.
  • Seriousness checks to improve the reliability of online surveys. Paper by Aust et al.
  • Detecting and deterring insufficient efforts in responding to surveys. Paper by Huang et al.
  • New paper by Meyer et al comparing several methods for detecting bots responding to online surveys. Matt Motyl developed a script to detect bots. There is an effort by Andy Wood along the same lines. Dennis et al argue that rather than bots, low quality answers are provided by humans using virtual private networks (VPNs) and offer solution for detecting them.
  • Randomly assigning people to different conditions in limesurvey.
  • A series of videos from the Beonline event covering issues in conducting online research: ensuring data quality, technical issues, recruiting participants…
  • To get into the mind of MTurkers, you can devise questionnaires. You can also check out this reddit page where they share their qualms.
  • OpenMTurk; open tool for managing MTurk studies.

Measurement

  • Clark and Watson’s (1995) paper on scale development.
  • Kenneth Bollen’s introduction to the concept of latent variable in psychology.
  • “”Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements”. Paper by Sam Parsons et al.
  • Measurement invariance, factor analysis and factorial invariance: 1993 paper by William Meredith proposing an excellent introduction to these concepts.  More recent beginner’s guide to these concepts (in evolutionary psychology, but it does not really matter) here by Wang et al (paywalled).
  • Why test-retest is not a measure of reliability or stability. Preprint by Wolf et al (seems useful and well done but not reviewed and I am not an expert on this).
  • Chapter by Malte Elson on question wording and item formulation
  • Excellent website with guidelines on designing rating scales.
  • Analyzing ordinal data with metric models: what could go wrong?  Paper by Liddell & Kruschke on problems with using metric models for likert scales.
  • Wonderful list of online measurement resources by Elko Fried and Jessica Flake
  • Helpful paper by He et al. on measurement artifacts and a solution to them: the coefficient of equivalence and stability (GCES)
  • Paper on the dangers of putting all items measuring the same construct in sequential order (and how to manage the reliability-validity tradeoff) by Clifton et al.
  • The wording effect refers to inconsistent findings on items that are reversed and “straight”. This paper by Abad et al.  proposes an approach to dealing with this problem.

Power estimation & Effect sizes

Effect Sizes

    • This paper by Funder & Ozer on how to approach effect sizes in psychological research is the best recent thing on the subject IMHO.
    • Clear introduction to estimating and reporting effect size by Daniël Lakens (Frontiers). Check Katherine Wood’s  online version (shiny): it is even more user-friendly. 
    • In the same vein, here is MOTE, a comprehensive online app by Erin Buchanan et al. for computing effect sizes based on a variety of designs and measurement levels. Play around with it. It’s fantastic!
    • Converting effect size (e.g., from d to eta square, etc). This page is great.
    • On the importance of small effects. Paper by Götz et al.
    • Great paper by Lovakov & Agadullina proposing guidelines for interpreting effect sizes.
    • Paper by Tosh et al. on  the Piranha effect. Large effects swimming in a small pond.

General Linear Model

    • A pratical primer by Perugini et al.  covering power analysis in simple experimental designs (including mediation, interactions involving continuous predictors, etc.). Will cover most of your needs and is supplemented by online material (excel sheets, etc.).
    • Superpower: online shiny app for computing power in ANOVA designs (simulation-based and exact) by Aaron Caldwell and Daniel Lakens.  There is also a R package. With manual here. Paper here.
    • G*Power, the free software that computes power for a variety of designs. The manual is here.
    • Declaredesign: Interactive R program that computes power based on your design (using simulations).
    • Computing power for interactions involving one continuous and one dichotomous variable. Online application designed by myself.
    • When there is more than one within subject factors, G*Power can’t compute power. The best solution is to run simulations but that requires programming skills. D’Amico et al. proposed a method using SPSS’s MANOVA procedure. Here is also another paper using this method for regression, correlation and simple anova designs.
    • Here is how to calculate power for a 3-way ANOVA in G*Power
    • Sequential data analysis. Great method for maximizing power and minimizing sample size at the same time. EJSP paper by Daniël Lakens.
    • How many participants do I need to test a moderation of the effect I found in my first study? Many. Post on Data Colada explaining this. 
    • A tutorial by myself on Determining sample size in social psychology (with tables). French.
    • Sample size planning adjusting for publication bias and uncertainty by Anderson et al.
    • Sample size planning for cognition and perception. Where Jeff Rouder explains that relatively small sample (around 30) can be OK if you use repeated measures.
    • Power2pp: Splendid R package that computes power for designs including t-tests, correlations, multiple regression, ANOVA, mediation, and logistic regression. Also covers mediation models (from Process models).
    • Wonderful tutorial by Liz Page-Gould on conducting simulations to estimate power for a moderated regression, multilevel model, and structural equation model.
    • Great tutorial by Julia Quantz on the same topic.

Mixed Models

    • Online calculator for power estimation in mixed models.
    • Powerlmm is a R package designed by Kristofer Magnusson for computing power in multilevel models. It can also help you assess whether your really need such a model.
    • Simr: Package for computer power in linear mixed models.
    • On the same topic, see this paper by Lane & Hennes on estimating sample size in multilevel models (in relationship research)
    • Pangea: A web applet that computes power for General Anova designs. By Jake Westfall. Thanks, Almudena Claassen, for pointing me to this!
    • For 2-levels mixed model, here is a shiny app calculator by You Marayama.

Other

    • Power analysis for mediation models: online app by Schoemann et al.
    • A free online book by Vanderbilts School et al. on solutions for conducting research with small sample sizes.
    • Pam Davis wrote on Twitter “I am putting together a resource of programs and code for doing statistical power analysis for all statistical procedures.” Here are the responses twittos provided.
    • Collaborative list of resources compiled by Gilad Feldman
    • Here is an online calculator for SEM by Christopher Preacher.

Qualitative methods / Text analysis

And here are resources on discourse analysis recommended by Theofilos Gkinopoulos:

Open science

        • How to Open Science: Great Great resource on all aspects of open science.
        • Our practical guide to transparency in psychological science (covering most aspects. Good to have a one-paper intro).
        • Compass to publish. Free tool (in French by the University of Liege to detect predatory journals.
        • Material of a workshop on replication by Patrick Forscher.
        • Our practical guide to transparency in psychological science: research planning /preregisration / data and material sharing/ publication. With extensive supplementary material. Includes an example “model” project on the OSF.

Preregistration

Displaying data

        • Using graphs instead of tables in political science: Great paper by Kastellec & Leoni showing how to replace tables by graphs. Applies to social psychology as well
        • In the same vein, alternatives to bar and line graphs. Useful paper by Tracey Weissgerber et al.
        • Plotting the confidence interval for regression estimates in R. That’s a problem I long struggled with and this discussion offers easy solutions for doing so.
        • Tools to enhance plots made by Ggplot with results of statistical tests. Developed by Indrajeet Patil.
        • Paper by McCabe et al with resources for plotting interactions optimally.
        • A list of free tools for creating more transparent figures for small datasets (well, the kind we mostly rely on in social psych).
        • University of Minnesota page showing examples of well formatted tables and graphs according to APA standards.
        • Seeing theory. Great website using splendid vizualizations to introduce to basic concepts in probability and stats.
        • Making it pretty: Plotting 2-way interactions with GGplot2. Nice tutorial & code (R).
        • Implementing Edward Tufte’s recommendations for cool looking graphs using R (Tufte is the absolute master of vizualization).
        • Guide de démarrage pour GGPlot. French.
        • The summarytools package (R). Provides summaries of variables in a data frame like this. Intro here.
        • Blog post by Adam Medcalf reviewing the various available tools to summarize data in R. Very useful!
        • Useful Tips from the @realscientists’ twitter account (a science illustrator for National Geographic) on improving figures (twitter thread)
        • Paper by Allen et al on raincloudplots (much better than bar graphs!) and resources (code, tutorials…) for drawing them.

Data management & preparation

        • Lectures on data management by Patrick Forscher.
        • General overview:  detecting & handling outliers by Leys et al.
        • Bakker & Witchers’s paper arguing against outlier removal.
        • Codebook,  a R package by Ruben Arslan that creates a codebook based on an SPSS file (haven’t tried it yet but looks great!).
        • Kai Horstmann provided very useful resources for constructing codebooks.
        • When you can’t access or share data (e.g., for ethical reasons), one possibility is to use a package that generates data having the same exact structure as the original ones. This is what the “synthpop” package does.

Testing

Statistical Inference (Fisher/Neymann-Pearson)

        • Jacob Cohen’s paper, Things I’ve learned so far. Essential reading that covers many of the core issues psychologists should be attuned to when conducting (inferential) statistical analyses.
        • Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Paper by Greenland et al. See also American Statistical Association’s statement on p values.
        • Understanding misconceptions about p values. Great blog post by Daniël Lakens.
        • Fisher, Neymann-Pearson or NHST? A tutorial for teaching data testing. Excellent paper by Perezgonzales et al.
        • One simple effect is significant, the other not but no interaction. Paper by Gelman on this.
        • Is it a problem to use parametric stats on likert scales when the sample size is low or the distribution far from normal? Usually not according to this paper by Geoff Norman.
        • p curve. Resources on using this method to detect p-hacking and other strange distributions of the p values in a programme of research.
        • Testing that the null is true without Bayes. Blog post by Daniël Lakens on Equivalence testing and the paper now in SPPS. See also a tutorial by Lakens et al. In the same vein, a concise open access paper by Etienne Quertemont (2011), “How to Statistically Show the Absence of an Effect”, covers equivalence testing, power analysis & use of confidence interval in a single paper. Useful for students especially.
        • A post by Heino Matti on false expectations about the relation between p values and sample size. Includes great vizualisations.
        • Aligning scientific reasoning and statistical inference: Short “Science” paper by Steven Goodman on misunderstandings in statistical inference and their impact on scientific progress.
        • Ever wondered about the meaning of degrees of freedom? Check out this excellent tutorial by Ron Dotsch (thanks, Rui Costa Lopes for the suggestion!) .
        • Beyond Statistics: Testing the null in mature sciences. Very informative paper by Morey et al.
        • Tukey’s 1991 paper on the philosophy of multiple comparisons. Where he rejects the dichotomous logic of significance testings and recommends the use of confidence intervals.
        • Russ Poldrack’s online stats textbook; Statistical thinking for the 21st century. Covers all basis statistical concepts to help researchers think straight about stats. With an emphasis on reproducibility.
        • Computing confidence intervals for proportions. Great page here.
        • Useful checklist to limit degrees of freedom in planning, conducting & reporting research and avoid p-hacking. By Witchers et al.
        • When to adjust alpha for multiple testing. Paper by Mark Rubin.
        • Paper by Halsey highlighting four alternatives to p-values.

Regression / General Linear Model (t-test, ANOVA, etc.)

t-test

        • Don’t use the Student t test anymore (Welsh test is better). If you are not convinced, check this paper by Delacre et al.
        • The perfect t-test. R program that reports the results of a t-test completely formatted, with graphs, tests of assumptions, etc. By Daniël Lakens.

Regression

ANCOVA

        • Great paper by Miller and Chapman on misunderstandings surrounding the interpretation of ANCOVA.
        • Using covariates when testing for interactions. Paper by Yzerbyt et al.

Contrast coding

        • Bob Abelson’s paper on contrast coding for testing interactions.
        • This excellent paper by Ista Zahn explains clearly how unequal sample size influence the estimation of contrast coefficients in multiple regression and what to do about it.

Mixed Models

        • Mixed Models: Introduction to treating stimuli as random factors and code for common statistical software by Westfall et al.
        • Follow-up on mixed models: Annual review chapter by the same authors addressing various research designs
        • Significance testing in lme4
        • Should you fit the “maximal model”? Parsimony in model construction. Paper by Bates et al.
        •  Centering predictors in mixed models. Paper by Enders & Tofighi.
        • Wonderful tutorial by Sommet and Morselli on multilevel logistic regression with scripts in R, Stata, MPlus and SPSS. Applied to Justin Bieber and a very fun read!
        • Paper by Fisher et al. showing that correlations between variables across individuals do not match correlations between these same variables (measured on many occasions) within individuals.
        • Explained variance measures for multilevel models. Paper by LaHuis et al.
        • Bodo Winter’s tutorials on mixed models using R.
        • This  very useful page displays the code for testing repeated measures designs in R using either the aov command or mixed models (lme and nlme).

Mediation & Moderation

        • Yzerbyt et al’s new recommendations for testing indirect effects in mediation models. In press in JPSP (06/2018): One should test the paths separately. This website designed by Cédric Batailler accompanies the R package that allows to apply this method easily to your data.
        • David Kenny’s simple and excellent mediation page.
        • Broader overview of mediation and moderation. By Judd et al (2014).
        • Interactions do not tell us when but also tell us how. Nice paper by Jaccoby & Sassenberg (2011).
        • A paper by Rik Pieters explaining when mediation analysis is warranted.
        • Why testing reverse mediation to check for directionality is a terrible idea:  Here  (Gollwitzer et al) and Here (Thoemmes)
        • Answers to the question “what’s the go-to-paper for why mediation analyses shouldn’t be reported as process evidence?” on twitter.
        • Why studying mediation is more difficult than it seems. Paper by Green et al.

Bayesian approaches

        • Zoltan Dienes’ very useful webpage on Bayesian stats for beginners (including online calculators).
        • How to get the most of nonsignificant results? Paper by Zoltan Dienes based on a a Bayesian approach.
        • Is there a free lunch in inference? Forceful advocacy of the Bayesian approach by Rouder et al. Very clear for nonspecialists.
        • Short intro to Bayesian stats with R examples by Fabian Dablander.
        • Tutorial for performing Bayesian t tests and ANOVAs, by Richard Morey.
        • Richard McElreath wrote a fantastic book “Statistical Rethinking” which, as it names suggests, invites readers to update their view of stats (using a Bayesian perspective). Even if you don’t have the book, he provides plenty of resources, including recorded lectures and slides, for pursuing this path.
        • BRMS is a R package designed by Paul Buerkner that tests linear mixed model using a bayesian approach. It uses the same syntax as lme4.

Structural Equation modeling

(see also the “measurement” section above for primers on measurement invariance and confirmatory factor analysis)

        • Paper by Goodboy & Kline: Statistical and practical concerns with research featuring structural equation modeling. Good primer on some of the errors you want to avoid!
        • All materials, including videos, of Sacha Epskamp’s online course on structural equation modeling. A trove!
        • “Invariance: What Does Measurement Invariance Allow us to Claim?” not much according to this paper by John Protzko.

Logistic Regression

Meta-Analysis

        • Metafun: excel spreadheet that allows you to easily implement meta-analysis in R using the “Metafor” package. Resources here.
        • Meta-analysis on SPSS. See Andy Field’s paper.
        • Tutorial on doing a meta-analysis in JASP.
        • Computing the meta-analytic effect size manually. Paper by Goh et al. (2016)
        • Nice post by Chris Holdgraf on designing and interpreting funnel plots.
        • How to conduct a mixed effect meta-analysis in R? If this question haunts you, check out this video by Sara Locatelli.
        • Screencast by Dan Quintana: How to perform a transparent meta-analysis. 

Social Networks

Statistical Software

SPSS

        • Annotated SPSS Output for Logistic Regression
        • Laerd stats: Online resources for learning Stats on SPSS. It’s not free but apparently, it’s worth the cost (I must say I prefer R but some people can’t live without SPSS).

R

Learning

General Reference

        • R for data science. A comprehensive and authoritative book on the subject. Online.
        • This post on the facebook page “R Users psychology” has a wonderful list of resources to learn R.

Markdown

        • Video tutorial by Michael C. Frank on using RMarkdown for integrating paper writing and data analysis. Handouts here.
        • RMarkdown: the definitive guide. Free.
        • Papaja: an add-on to RMarkdown that formats papers in line with APA style requirements. Fantastic work by Frederik Aust. And if you’d rather watch a video tutorial, here is one by Nick Fox.
        • Guide démarrage en français par Claire Della Vedova.
        • citr: an addin in Rstudio (designed by Frederik Aust) that allows to insert Markdown citations based on a library you specify. Super easy. It supports the open source reference manager Zotero.

Other

        • JASP: a free  R based statistical software that also performs bayesian tests. Super easy to use. Great for teaching especially. See a vast amount of resources on using JASP to do Bayesian stats here. Here is a practical primer by Hu et al.
        • Jamovi: Same logic. Even better. Open source.

Datasets

        • Open Stats Lab: Freely available datasets to play around. It comes with the corresponding article. Ideal for teaching purposes.
        • This response to a tweet by Vicky Boykis has plenty of suggestions for freely available datasets to use in R or elsewhere.
        • List of freely available datasets (editable). By Cameron Brick.

Ethics

Other stuff

        • Undergraduate statistics with JASP. Resources compiled by Erin Buchanan.
        • Statistics of Doom: a treasure trove. Erin Buchanan’s classes on stats are well organized and cover everything you will need.
        • An open access stat introductory textbook with great animated vizualizations (coded in R & ggplot) by Matt Crump et al.
        • Shiny Web Apps for designing experiments and analysing data
        • Rpsychologist: All kinds of vizualisations of common statistical procedures. Splendid for pedagogical purposes especially.
        • Paper  by Butts et al. on the source of common errors in the interpretation of cutoff criteria for widely used stats (including .70 for Cronbach’s alpha)
        • Blog post by Dick Morey on the dangers of averaging data.
        • On a similar topic, paper by Colin Leach on how to integrate the person with macro-level processes (and look differently at regression plots).
        • Paper by LeBel & al summarizing criticisms of open science and advocacy of high powered research movement in social psych and addressing them. Useful to find all the important recent references.
        • * Statcheck: Online tool to check whether there any errors in your paper. Just upload a PDF.
        • SPRITE: method developed by James Heathers et al to reconstruct datasets based on available means, SDs and Ns. A great tool for detecting errors. The shiny app  is here.
        • Learning to use GitHub. Module 5 of an excellent Open Science MOOC.
        • Githbub for R users: Excellent tutorial by Jenny Brian.
        • ESSurvey: RPackage for downloading and analyzing data from the European Social Survey.
        • Paul Minda’s lab manual (OSF)
        • What is replication? Paper by Nosek et al.
        • A complete research methods course for undergraduates by Katie Corker.
        • How to do a systematic review? Nice paper with suggestions for best practices.
        • * Recite: Wonderful tools that checks whether your citations match your references.
        • Tutorial on exploratory research by Szabelska et al.
        • Internet archive scholar: Search engine for archived scientific papers that you may not find on google scholar.

 

By Cescup | 5 July 2016 | Statistics | 0 Comments |

Previous

Comment définir la taille d'un échantillon?
Next

Ressources "Comprendre la psychologie en tant que sciences" de Zoltan Dienes

Leave a Reply

Your email address will not be published. Required fields are marked *