Reliability and Calibrated Prediction #491

NathanielF · 2023-01-06T13:05:17Z

This will be an example notebook demonstrating the techniques of survival analysis used in reliability context with a focus on the predictive distribution and calibrated prediction intervals from a Bayesian and MLE perspective. Relates to issue: #474

Adding this draft version here, because i'm having a little trouble with the pre-commit checks

Notebook follows style guide https://docs.pymc.io/en/latest/contributing/jupyter_style.html
PR description contains a link to the relevant issue:
- a tracker one for existing notebooks (tracker issues have the "tracker id" label)
- or a proposal one for new notebooks
Check the notebook is not excluded from any pre-commit check: https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml

…n branch Signed-off-by: Nathaniel <[email protected]>

review-notebook-app · 2023-01-06T13:05:22Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

NathanielF · 2023-01-06T13:23:01Z

Hi @drbenvincent and @OriolAbril

I think this is ready for review now. It's longer than i anticipated and i've cut out the idea to compare against conformal prediction as there is already quite a bit of detail in there.

The broader structure has three parts:

In the first part i'm just introducing time-to-failure data and the descriptive statistical analogues of survival and cdf functions. I follow a trajectory of first showing how we can analyse this kind of data with MLE or bootstrap style inference and then show on a trickier example with sparser data why the ability to add Bayesian priors to help calibrate the risk the profile is useful. Then i conclude with a "lets all be friends" kind of ending.

I'm not sure who is best placed to review, but since i deal with censored data and I know @drbenvincent recently updated the censored data notebook, i thought you might find it interesting? Curious about what you think too @OriolAbril? Happy to take any feedback on the content or structure...

…ied priors. Signed-off-by: Nathaniel <[email protected]>

OriolAbril

I have commented on the myst file, but the changes should be done in ipynb and then use pre-commit to update myst one.

General comment: I would use https://myst-nb.readthedocs.io/en/latest/render/hiding.html to hide the source of some of the code. i.e. the stringIO ones can be hidden, we would only need to split them so the df_name to show the rendered table is in its own cell right below (not hidden so readers can follow the notebook and get the variable name even without expanding the cell). Some of the plots I think could also have their source hidden.

OriolAbril · 2023-01-09T17:23:47Z

examples/case_studies/reliability_and_calibrated_prediction.myst.md

+We'll use the posterior predictive distribution of the uniformative model. We show here how to derive the uncertainty in the estimates of the 95% prediction interval for the number of failures in a time interval.
+
+```{code-cell} ipython3
+def PI_failures(joint_draws, lp, up, n_at_risk):


I'll update the code to use xarray-einstats, the inputs are xarray dataset and scalars IIUC, so there should be no need for looping in order to use scipy distributions. I can also try rerunning the notebook and finding other similar instances.

On this one... i think i kind of disagree. While it's not the most efficient way to do this operation, i wanted the parallel with the bootstrapping looping procedures to be clear. We're instantiating in a loop a reasonable model fit to the data in each iteration of the loop, and seeing the variation induced in downstream statistics... i think pedagogically this is simpler than hiding the similarity of the procedure in more efficient code....

Oops I think I misread this comment yesterday, did you want to try and streamline the loop? My only point was that I'd like to keep the analogy with the bootstrap procedure clear....

But I think I could just add more text explaining that too.

Ok, i've added some more text to highlight the relationship between the bootstrap and posterior predictive distributions if you want to work some xarray magic, that'd be cool! Thanks!

Updated, and found a bug in xarray-einstats while doing it 😄 (I have fixed it already)

Is it this one:

I updated to my xarray einstats version, but still getting an error when i try to re-run your new function?

Can recreate with just this call:

Oh!!!! Sorry, i see the issue, the latest change is not on pip!!!! Sorry, slow this morning.

Ehm, that's cool. I'm happy with the changes and you can merge if you want. But for reproducibility should we be concerned that the user can't easily install the required packages?

I still have one PR I to get into 0.5 release, but I plan to release it soon. We can wait until I actually make the release one of these days, with the watermark having the 0.5.0dev flag though I don't think it is a deal breaker to merge it even before that.

OK, cool! Let's do it. Thanks again for all your help on this one! I'm happy with it now if you are.

examples/case_studies/reliability_and_calibrated_prediction.myst.md

…hide data import and plotting funcs Signed-off-by: Nathaniel <[email protected]>

NathanielF · 2023-01-09T19:12:37Z

Thanks so much for the review. @OriolAbril. I've quibbled with one of the requests but i think i've addressed the rest. I've added hide-input tags to a bunch of the data loads and some of the more involved plotting functions.

…cer in thumbnail Signed-off-by: Nathaniel <[email protected]>

…diction intervals with posterior predictive intervals Signed-off-by: Nathaniel <[email protected]>

OriolAbril

I reran the notebook locally to update that sampling snippet. And did a couple extra changes that I have commented here in the review. Let me know what you think, they are all quite minor except for the sampling one.

examples/case_studies/reliability_and_calibrated_prediction.myst.md

OriolAbril · 2023-01-15T20:13:50Z

examples/case_studies/reliability_and_calibrated_prediction.myst.md

-joint_draws = az.extract_dataset(idata, group="posterior", num_samples=1000)[
-    ["alpha", "beta"]
-].to_dataframe()
+joint_draws = az.extract(idata, num_samples=1000)[["alpha", "beta"]]


The fact that was a dataframe wasn't being used anywhere, so I changed that because now below we take advantage of the fact joint_draws is an xarray dataset.

That's fair.

OriolAbril · 2023-01-15T20:15:09Z

examples/case_studies/reliability_and_calibrated_prediction.myst.md

+We'll use the posterior predictive distribution of the uniformative model. We show here how to derive the uncertainty in the estimates of the 95% prediction interval for the number of failures in a time interval.
+
+```{code-cell} ipython3
+def PI_failures(joint_draws, lp, up, n_at_risk):


Updated, and found a bug in xarray-einstats while doing it 😄 (I have fixed it already)

NathanielF · 2023-01-16T09:34:16Z

Thanks @OriolAbril I'm fine with all the changes you made, i understand the logic of the xarray-einstats version too. I think the code is fine, just using the dev version of xarray-einstats makes reproducibility harder. We could add a note to the effect that this uses a dev version of xarrray. What do you think?

Signed-off-by: Nathaniel <[email protected]>

Fixed typo on myst too

NathanielF · 2023-01-16T10:32:18Z

Just fixing a typo at the beginning.

NathanielF · 2023-01-16T19:35:51Z

Woop, woop! Thanks 😊!

[Reliability Bayesian pymc-devs#474] Adding notebooks and bib on clea…

71e4a35

…n branch Signed-off-by: Nathaniel <[email protected]>

NathanielF marked this pull request as ready for review January 6, 2023 13:23

[Reliability Bayesian pymc-devs#474] added cost function plot and var…

8611c5b

…ied priors. Signed-off-by: Nathaniel <[email protected]>

NathanielF mentioned this pull request Jan 9, 2023

Bayesian Methods for Reliability Data #474

Closed

OriolAbril reviewed Jan 9, 2023

View reviewed changes

[Reliability Bayesian pymc-devs#474] updated with review comments to …

fe06eed

…hide data import and plotting funcs Signed-off-by: Nathaniel <[email protected]>

NathanielF added 2 commits January 9, 2023 19:33

[Reliability Bayesian pymc-devs#474] updated final plot to display ni…

2bf64cd

…cer in thumbnail Signed-off-by: Nathaniel <[email protected]>

[Reliability Bayesian pymc-devs#474] added text to link bootstrap pre…

d1854ac

…diction intervals with posterior predictive intervals Signed-off-by: Nathaniel <[email protected]>

NathanielF requested a review from OriolAbril January 12, 2023 09:29

update bayesian predictions to use einstats

a03ca61

OriolAbril reviewed Jan 15, 2023

View reviewed changes

NathanielF added 2 commits January 16, 2023 09:59

[Reliability Bayesian pymc-devs#474] fixed minor typo

89424d3

Signed-off-by: Nathaniel <[email protected]>

Update reliability_and_calibrated_prediction.myst.md

e3c3407

Fixed typo on myst too

OriolAbril approved these changes Jan 16, 2023

View reviewed changes

OriolAbril merged commit 462700a into pymc-devs:main Jan 16, 2023

NathanielF deleted the reliability_calibration_clean branch January 16, 2023 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reliability and Calibrated Prediction #491

Reliability and Calibrated Prediction #491

NathanielF commented Jan 6, 2023 •

edited

Loading

review-notebook-app bot commented Jan 6, 2023

NathanielF commented Jan 6, 2023

OriolAbril left a comment

OriolAbril Jan 9, 2023 •

edited

Loading

NathanielF Jan 9, 2023

NathanielF Jan 10, 2023

NathanielF Jan 12, 2023

OriolAbril Jan 15, 2023

NathanielF Jan 16, 2023

NathanielF Jan 16, 2023

NathanielF Jan 16, 2023

OriolAbril Jan 16, 2023

NathanielF Jan 16, 2023

NathanielF commented Jan 9, 2023

OriolAbril left a comment

OriolAbril Jan 15, 2023

NathanielF Jan 16, 2023

OriolAbril Jan 15, 2023

NathanielF commented Jan 16, 2023 •

edited

Loading

NathanielF commented Jan 16, 2023 •

edited

Loading

NathanielF commented Jan 16, 2023

Reliability and Calibrated Prediction #491

Reliability and Calibrated Prediction #491

Conversation

NathanielF commented Jan 6, 2023 • edited Loading

review-notebook-app bot commented Jan 6, 2023

NathanielF commented Jan 6, 2023

OriolAbril left a comment

Choose a reason for hiding this comment

OriolAbril Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanielF commented Jan 9, 2023

OriolAbril left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanielF commented Jan 16, 2023 • edited Loading

NathanielF commented Jan 16, 2023 • edited Loading

NathanielF commented Jan 16, 2023

NathanielF commented Jan 6, 2023 •

edited

Loading

OriolAbril Jan 9, 2023 •

edited

Loading

NathanielF commented Jan 16, 2023 •

edited

Loading

NathanielF commented Jan 16, 2023 •

edited

Loading