Add case study for Bayesian A/B Testing

[tcuongd]

Description

Hi! I'd like to add a case study notebook to provide an introduction to A/B test analysis with Bayesian models. The models implemented are based on this paper. Super keen for feedback / whether or not you think this is useful to add to the examples directory.

If you guys find it interesting, I'm keen to write more notebooks to go into more complicated topics like early stopping of A/B tests, power analyses, and more complex models and priors. I think there's a lot of practical context here beyond the usual textbook examples that would be valuable to readers working in online software.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[twiecki]

Thanks @tcuongd! Bayesian AB testing is a super common use case for pymc3 but so far there were no case studies so this is super relevant. Definitely interested in more in-depth stuff.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

ericmjl commented on 2021-05-24T01:51:15Z
----------------------------------------------------------------

It's probably a good idea to show the connection of A/B testing to traditional design of experiments. A/B testing is merely a fancy name for case vs. control experiment designs, where one of the groups, say "A", is the control group while the other ("B") is the case group; it's a staple experimental design in any of the experimental sciences (e.g. life sciences).

[ericmjl]

Nice stuff, @tcuongd 😄. Just one comment from me, above!

[tcuongd]

Thanks for the feedback guys :) I've changed the introduction to show a clearer link between A/B testing and controlled experiments.

I've also fixed some bugs with the revenue model example.

Thanks!

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

OriolAbril commented on 2021-05-25T09:49:39Z
----------------------------------------------------------------

Would it be worth it to also show this plot? Or do you plan to cover things related to this in another notebook? Having a link to the other notebook once it's added would be prefectly fine too. I think there should be at least a link because many readers of the notebook will probably not be too familiar with pymc (or Bayes either) and may not know what to do with this advise

tcuongd commented on 2021-05-25T13:08:41Z
----------------------------------------------------------------

Ah yes, good idea! I think we might as well show the plots here.

[OriolAbril]

The notebook is great, and further notebooks on the topic would be amazing 🚀

[tcuongd]

Ah yes, good idea! I think we might as well show the plots here.

---

View entire conversation on ReviewNB

[tcuongd]

@OriolAbril Could you please take another look? Thank you!

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

OriolAbril commented on 2021-05-25T16:20:10Z
----------------------------------------------------------------

How about plotting them side by side?

ax = az.plot_posterior(
    scenario_value_2, var_names=["theta_reluplift_1", "reciprocal_lam_reluplift_1"], textsize=10, kind="hist"
)
ax[0].set_title(f"Conversion Rate Uplift B, True Uplift = {(0.04 / 0.05 - 1):.2%}", fontsize=10)
ax[0].axvline(x=0, color="red")
ax[1].set_title(
    f"Revenue per Converting Visitor Uplift B, True Uplift = {(25 / 20 - 1):.2%}", fontsize=10
)
ax[1].axvline(x=0, color="red");

They would probably also benefit from a couple sentences restating how the model retrieves the less but larger purchases.

[tcuongd]

They would probably also benefit from a couple sentences restating how the model retrieves the less but larger purchases.

Ah side-by-side is a good idea, and I didn't know you could do that directly with az.plot_posterior, nice! I've made the changes :)

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

OriolAbril commented on 2021-05-26T19:57:04Z
----------------------------------------------------------------

Line #12.    axs[1].axvline(x=0, color="red");

I would add a colon here to suppress the extra "lines" output

[OriolAbril]

That last comment is a very minor nit. IMO, it can be merged right after it's fixed, everything else looks great

[tcuongd]

That last comment is a very minor nit. IMO, it can be merged right after it's fixed, everything else looks great

Thanks Oriol! Have fixed :)