misc updates

Author: drbenvincent

examples/howto/hypothesis_testing.ipynb

@@ -60,7 +60,7 @@
     "  * Posterior probability statements\n",
     "  * Highest Density Intervals (HDIs)\n",
     "  * Regions of Practical Equivalence (ROPE)\n",
-    "  * Bayes Factors using ArviZ's `plot_bf` function\n",
+    "  * Bayes Factors\n",
     "\n",
     "We'll work through a scenario where we want to know if the mean of some metric (e.g., monthly profit) is different from zero."
    ]
@@ -77,6 +77,22 @@
     ":::"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "6292d775",
+   "metadata": {},
+   "source": [
+    "### Parameter estimation vs model comparison\n",
+    "\n",
+    "The Bayesian evaluation of null values can proceed in two distinct ways (see {cite:t}`kruschke2011bayesian`):\n",
+    "\n",
+    "#### Parameter estimation\n",
+    "The parameter estimation approach considers a model where the parameter is allowed to vary (which includes the null value). We then compute the posterior distribution of this value and come up with some kind of decision rule which determines if we accept or reject the null value.\n",
+    "\n",
+    "#### Model comparison\n",
+    "Two competing models are considered. The first model assumes that the null value is true, or fixed, and the second model allows a range of values. The models are compared to see which is more supported by the data. An example would be in assessing if a coin is fair (null hypothesis) or biased (alternative hypothesis) - we would set up a model where the coin has a fixed probability of heads (0.5) and another model where the probability of heads is a free parameter. Readers are referred to the PyMC examples focussing on {ref}`pymc:model_comparison` for more details."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "5c268654",
@@ -525,6 +541,14 @@
     "az.plot_posterior(idata, var_names=[\"mu\"], rope=rope, figsize=(14, 3));"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "5fc84f7a",
+   "metadata": {},
+   "source": [
+    "The intuition we can get from this is that if the ROPE is narrow, we would require quite a high level of precision to accept the null hypthesis. The posterior distribution would have to be very tightly centered around the null value to have a large probability of being within the ROPE."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "46cf08f2",
@@ -538,11 +562,15 @@
    "id": "de465219",
    "metadata": {},
    "source": [
-    "{cite:t}`kruschke2018rejecting` outlines the HDI+ROPE decision rule, which is summarised in the figure taken from that paper:\n",
+    "{cite:t}`kruschke2018rejecting` outlines the HDI+ROPE decision rule, which is summarised in the figure taken from that paper. Namely:\n",
+    "\n",
+    "* **Accept the null value**: If the HDI falls entirely within the ROPE. The HDI does not need to include the null value.\n",
+    "* **Reject the null value**: If the HDI falls entirely outside the ROPE. \n",
+    "* **Remain undecided**: If the HDI is partially or fully outside the ROPE.\n",
     "\n",
-    "![](hdi_plus_rope_decision_rule.png)\n",
+    "In our case, looking back at our posterior + ROPE plot above, we would remain undecided because the HDI does not fall entirely outside nor within the ROPE.\n",
     "\n",
-    "In our case, we would remain undecided because the HDI does not fall entirely outside nor within the ROPE."
+    "![](hdi_plus_rope_decision_rule.png)"
    ]
   },
   {
@@ -602,6 +630,14 @@
     "We can see that the probability of $\\mu=0$ has gone _down_ after observing the data. This is reflected in the value of $BF_{01}=0.54$ in that it is less than 1."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "4283bdd3",
+   "metadata": {},
+   "source": [
+    "Readers are referred to {ref}`Bayes_factor` for a more detailed look at Bayes Factors."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "650f409c",

examples/howto/hypothesis_testing.myst.md

@@ -45,7 +45,7 @@ In this tutorial, we'll use PyMC to:
   * Posterior probability statements
   * Highest Density Intervals (HDIs)
   * Regions of Practical Equivalence (ROPE)
-  * Bayes Factors using ArviZ's `plot_bf` function
+  * Bayes Factors
 
 We'll work through a scenario where we want to know if the mean of some metric (e.g., monthly profit) is different from zero.
 
@@ -59,6 +59,18 @@ However, in real-world scenarios—such as policy-making, resource allocation, o
 
 +++
 
+### Parameter estimation vs model comparison
+
+The Bayesian evaluation of null values can proceed in two distinct ways (see {cite:t}`kruschke2011bayesian`):
+
+#### Parameter estimation
+The parameter estimation approach considers a model where the parameter is allowed to vary (which includes the null value). We then compute the posterior distribution of this value and come up with some kind of decision rule which determines if we accept or reject the null value.
+
+#### Model comparison
+Two competing models are considered. The first model assumes that the null value is true, or fixed, and the second model allows a range of values. The models are compared to see which is more supported by the data. An example would be in assessing if a coin is fair (null hypothesis) or biased (alternative hypothesis) - we would set up a model where the coin has a fixed probability of heads (0.5) and another model where the probability of heads is a free parameter. Readers are referred to the PyMC examples focussing on {ref}`pymc:model_comparison` for more details.
+
++++
+
 ## Setting up the example
 
 Rather than focus on a complex example, we'll pick a trivial one were we are simply estimating the mean of a single variable. This will allow us to focus on the hypothesis testing. The important thing is what we do with our MCMC samples, not the particulars of the model.
@@ -179,15 +191,23 @@ Third time in a row, `arviz` has our back and can plot the ROPE and HDIs.
 az.plot_posterior(idata, var_names=["mu"], rope=rope, figsize=(14, 3));
 ```
 
+The intuition we can get from this is that if the ROPE is narrow, we would require quite a high level of precision to accept the null hypthesis. The posterior distribution would have to be very tightly centered around the null value to have a large probability of being within the ROPE.
+
++++
+
 ### HDI+ROPE decision criteria
 
 +++
 
-{cite:t}`kruschke2018rejecting` outlines the HDI+ROPE decision rule, which is summarised in the figure taken from that paper:
+{cite:t}`kruschke2018rejecting` outlines the HDI+ROPE decision rule, which is summarised in the figure taken from that paper. Namely:
 
-![](hdi_plus_rope_decision_rule.png)
+* **Accept the null value**: If the HDI falls entirely within the ROPE. The HDI does not need to include the null value.
+* **Reject the null value**: If the HDI falls entirely outside the ROPE. 
+* **Remain undecided**: If the HDI is partially or fully outside the ROPE.
 
-In our case, we would remain undecided because the HDI does not fall entirely outside nor within the ROPE.
+In our case, looking back at our posterior + ROPE plot above, we would remain undecided because the HDI does not fall entirely outside nor within the ROPE.
+
+![](hdi_plus_rope_decision_rule.png)
 
 +++
 
@@ -212,6 +232,10 @@ We can see that the probability of $\mu=0$ has gone _down_ after observing the d
 
 +++
 
+Readers are referred to {ref}`Bayes_factor` for a more detailed look at Bayes Factors.
+
++++
+
 ## Summary
 
 **Posterior Probability Statements**