Journal of Experimental Social Psychology
Journal of Experimental Social Psychology
5 years ago

Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation

The practice of using covariates in experimental designs has become controversial. Traditionally touted by statisticians as a useful method to soak up noise in a dependent variable and boost power, the practice recently has been recast in a negative light because of Type I error inflation. But in order to make informed decisions about research practices like this one, researchers need to know more about the actual size of the benefits and costs of these practices. In a series of simulations, we compared the Type I error rates and power of two analytic practices that researchers might use when confronted with an unanticipated, independent covariate. In the baseline practice, a researcher only analyzes the effect of the manipulation on the dependent variable; in the flexible-covariate practice, she analyzes both the effect of the manipulation on the dependent variable and the effect adjusting for the unanticipated covariate. We show that the flexible-covariate (vs. baseline) practice inflates Type I error by a small amount, and that it boosts power substantially under certain circumstances. The flexible-covariate practice tends to be most beneficial when the covariate is strongly correlated with the dependent variable in the population, and when the experimental design would have been only moderately powered (40%–60%) without including the covariate in the analysis. We offer concrete recommendations for when and how to use independent covariates in experimental designs, and contextualize our findings within the movement toward quantifying trade-offs in choosing among research practices and optimizing the choice of practice within a given research context.

Publisher URL: www.sciencedirect.com/science

DOI: S0022103116308095

You might also like
Discover & Discuss Important Research

Keeping up-to-date with research can feel impossible, with papers being published faster than you'll ever be able to read them. That's where Researcher comes in: we're simplifying discovery and making important discussions happen. With over 19,000 sources, including peer-reviewed journals, preprints, blogs, universities, podcasts and Live events across 10 research areas, you'll never miss what's important to you. It's like social media, but better. Oh, and we should mention - it's free.

  • Download from Google Play
  • Download from App Store
  • Download from AppInChina

Researcher displays publicly available abstracts and doesn’t host any full article content. If the content is open access, we will direct clicks from the abstracts to the publisher website and display the PDF copy on our platform. Clicks to view the full text will be directed to the publisher website, where only users with subscriptions or access through their institution are able to view the full article.