Comments on: Frequentist vs. Bayesian approach in A/B testing https://www.dynamicyield.com/lesson/bayesian-testing/ Wed, 31 Jan 2024 06:44:00 +0000 hourly 1 By: Georgi Georgiev https://www.dynamicyield.com/lesson/bayesian-testing/#comment-669 Wed, 28 Mar 2018 10:02:00 +0000 https://www.dynamicyield.com/?p=17150#comment-669 Hi Idan,

To state that the simple p-value is a “”convoluted term”” is quite interesting, especially given that you compare them to Bayesian HPDR or credible intervals stating they are easier to interpret. Prof. Mayo does a great job at dispensing with the myth of the simplicity of reporting posteriors or other Bayesian statistics here: https://errorstatistics.com/2013/07/06/a-bayesian-bear-rejoinder-practically-writes-itself-2/

I provide a more visual illustration of the difference between the two in my post on Bayesian A/B testing, here it is for convenience:

Frequentist p-value: P(x > Δe ; null hypothesis)

Bayesian posterior: P(x > Δe | null hypothesis) · P(null hypothesis) / P (x > Δe)

As you can see the equation for the p-value is mathematically just a part of the posterior estimation. Claiming it is easier to understand a more complex mathematical method is curious, to say the least.

Other claims about frequentist approaches are interesting as well: “”It requires a known baseline and reaching a predefined sample size before we’re allowed to look at the data and draw conclusions.”” – only if you are using methods developed in the 1930s, ignoring all the progress from the past 70 years. There are well developed and not so hard to implement sequential procedures (certainly easier than many Bayesian ones) that allow for flexibility in examining data as it gathers via alpha and beta stopping boundaries. An example would be the AGILE statistical method I’ve proposed, among other possible solutions.

Stating that you don’t need to fix your sample size in advance and that peeking is allowed with Bayesian approaches is not true (you do make a point to the revers with the “”with caution”” link, but it doesn’t really explain how caution should be applied, that is – how to plug in the information about the stopping rule into the prior. The myth of the immunity to optional stopping of Bayesian approaches is widespread – it would be nice to not contribute any further to it.

Best,
Georgi

]]>
By: Steven https://www.dynamicyield.com/lesson/bayesian-testing/#comment-668 Mon, 12 Mar 2018 16:01:00 +0000 https://www.dynamicyield.com/?p=17150#comment-668 In your article you mention the following:
Then, the Bayesian begins to measure heights of specific American citizens, and with each measurement updates the distribution to become a bit more “bell shaped” around the average height measured so far. As more data is collected, the “bell” becomes sharper and more concentrated around the measured average height.

What is meant by “specific” American citizens. Does a Bayesian measure a citizen from each size until a sample size of 1000 citizens is reached? Or take 1000 random citizens?

]]>