In a recent Nature paper, Marek, Tervo-Clemmens (T-C) et al. evaluated the effects of sample size on univariate and multivariate Brain-Wide Association Studies (BWAS) in three large-scale neuroimaging datasets and came to the general conclusion that

BWAS reproducibility requires samples with thousands of individuals.

But is this always the case? Let’s have a closer look at the replicability of functional connectivity-based multivariate BWAS in the Human Connectome Project, with N=500:

Replicability and predictive effect size at N=500

Interactive figures, no motion-bias (p>0.05 for all, tested with mlconfound). See analysis notebook.

Appearantly, BWAS can be replicable with moderate sample sizes in many cases…

If you wonder how to interpret these findings in light of Marek, T-C et al.’s results:

1. Check out our commentary

“Multivariate BWAS can be replicable with moderate sample sizes”

by Tamas Spisak, Ulrike Bingel & Tor Wager, Nature, 615, E4–E7 (2023) doi: 10.1038/s41586-023-05745-x

2. Watch the video abstract

3. Read more below

Summary

• Brain-Wide Association Studies (BWAS) correlate individual differences in phenotypic traits with measures of brain structure and function.
• In a recent paper, Marek, Tervo-Clemmens (T-C) et al. evaluated the effects of sample size on univariate and multivariate BWAS in three large-scale neuroimaging datasets and came to the general conclusion that “BWAS reproducibility requires samples with thousands of individuals”.
• Marek, T-C, et al. find that multivariate BWAS provide “inflated in-sample associations” that often fail to replicate (i.e., are underpowered) unless thousands of participants are included. This implies that effect size estimates from the discovery sample are necessarily inflated.
• In our ‘Matter Arising’ commentary, we distinguish between the effect size estimation method (in-sample vs. cross-validated) and the sample (discovery vs. replication), and show that with appropriate cross-validation the in-sample “inflation” Marek, T-C, et al. report in the discovery sample can be entirely eliminated.
• With additional analyses, we demonstrate that multivariate BWAS effects in high quality datasets can be replicable with substantially smaller sample sizes in many cases. Specifically, with a standard multivariate predictive model, functional connectivity-based BWAS is commonly replicable with N=75-500.

Take Home Messages and Recommendations

• Replicability depends on effect size:
  • Only effects that explain less than 1% of variance require 1000s
  • Effect sizes depend on the amount, quality, homogeneity and reliability of both brain and phenotypic measures
• Multivariate BWAS (predictive modelling) can provide high effect sizes:
  • Go multivariate
  • Follow methodological recommendations
  • Aim at traits (cognition, personality) rather than states (e.g. emotion)
  • Incorporate within-person variation in symptoms or behavior to improve between person predictions
  • Focus on larger effects, that are usueful for making individual predictions about individuals
• Validate Twice (internally and externally):
  • Perform internal validation during model discovery (e.g. cross-validation)
  • Finalize your model (including data preprocessing) and pre-register further validation
  • Externally validate on independent (prospective) data.
• Large samples are still essential:
  • to maximize predictive performance
  • to evaluate
    • generalizability (e.g. out-of-center, out-of-context)
    • confounding bias (e.g. motion; can be tested with mlconfound)
    • fairness (across subpopulations)

Links

• The original paper: Marek, S., Tervo-Clemmens, B., Calabro, F.J. et al. Reproducible brain-wide association studies require thousands of individuals. Nature 603, 654–660 (2022)
• Our commentary: Spisak, T., Bingel, U. & Wager, T.D. Multivariate BWAS can be replicable with moderate sample sizes. Nature 615, E4–E7 (2023). https://doi.org/10.1038/s41586-023-05745-x
• Analysis code on github

Other papers in the topic