arXiv Statistics Theory
arXiv Statistics Theory
Preprint
4 years ago

Dealing With Ratio Metrics in A/B Testing at the Presence of Intra-User Correlation and Segments. (arXiv:1911.03553v2 [stat.AP] UPDATED)

Keyu Nie, Yinfei Kong, Ted Tao Yuan, Pauline Berry Burke
We study ratio metrics in A/B testing at the presence of correlation among observations coming from the same user and provides practical guidance especially when two metrics contradict each other. We propose new estimating methods to quantitatively measure the intra-user correlation (within segments). With the accurately estimated correlation, a uniformly minimum-variance unbiased estimator of the population mean, called correlation-adjusted mean, is proposed to account for such correlation structure. It is proved theoretically and numerically better than the other two unbiased estimators, naive mean and normalized mean (averaging within users first and then across users). The correlation-adjusted mean method is unbiased and has reduced variance so it gains additional power. Several simulation studies are designed to show the estimation accuracy of the correlation structure, effectiveness in reducing variance, and capability of obtaining more power. An application to the eBay data is conducted to conclude this paper.

Publisher URL: http://arxiv.org/abs/1911.03553

DOI: arXiv:1911.03553v2

You might also like
Discover & Discuss Important Research

Keeping up-to-date with research can feel impossible, with papers being published faster than you'll ever be able to read them. That's where Researcher comes in: we're simplifying discovery and making important discussions happen. With over 19,000 sources, including peer-reviewed journals, preprints, blogs, universities, podcasts and Live events across 10 research areas, you'll never miss what's important to you. It's like social media, but better. Oh, and we should mention - it's free.

  • Download from Google Play
  • Download from App Store
  • Download from AppInChina

Researcher displays publicly available abstracts and doesn’t host any full article content. If the content is open access, we will direct clicks from the abstracts to the publisher website and display the PDF copy on our platform. Clicks to view the full text will be directed to the publisher website, where only users with subscriptions or access through their institution are able to view the full article.