Gaussian Process inconsistent results with find_MAP

[danhphan]

Description of your problem

Hi when running GP find_MAP in PyMC version4 seems to return different results compared to PyMC3.

Example 1: The GP salmon notebook

Pymc v4: https://github.com/danhphan/gp_experiments/blob/main/examples/gp_salmon_pymc_v4.ipynb

{'ρ_log__': array(1.60943791),
 'η_log__': array(1.60943791),
 'σ_log__': array(3.91202301),
 'ρ': array(5.),
 'η': array(5.),
 'σ': array(50.)}

Pymc v3: https://github.com/danhphan/gp_experiments/blob/main/examples/gp_salmon_pymc_v3.ipynb

{'ρ_log__': array(-7.95467461),
 'η_log__': array(-31.66553291),
 'σ_log__': array(3.96207137),
 'ρ': array(0.00035102),
 'η': array(1.76943166e-14),
 'σ': array(52.56609719)}

Example 2: GP-Kron

PyMC V4: https://github.com/danhphan/gp_experiments/blob/main/examples/GP-Kron.ipynb

{'ls1_log__': array(0.),
 'ls2_log__': array(0.),
 'eta_log__': array(0.69314718),
 'sigma_log__': array(0.69314718),
 'ls1': array(1.),
 'ls2': array(1.),
 'eta': array(2.),
 'sigma': array(2.)}

PyMC v3: https://github.com/pymc-devs/pymc-examples/blob/main/examples/gaussian_processes/GP-Kron.ipynb

{'ls1_log__': array(0.3382134),
 'ls2_log__': array(0.00040654),
 'eta_log__': array(0.9907769),
 'sigma_log__': array(-1.39195968),
 'ls1': array(1.40243975),
 'ls2': array(1.00040662),
 'eta': array(2.69332612),
 'sigma': array(0.24858767)}

The detailed versions of main packages are printed at the end of each notebook. Thanks

• Python Version: 3.9
• Operating system: Ubuntu 20.04
• How did you install PyMC/PyMC3: (conda/pip) pip

[ricardoV94]

@danhphan can you show the logp (without jacobian) of the model at those points to confirm that v4 is actually the one doing the worse job?

[danhphan]

Hi, I am not sure how to do that.

It should be a lot quicker if you download this notebook https://github.com/danhphan/gp_experiments/blob/main/examples/gp_salmon_pymc_v4.ipynb and check the logp on your side :) . I guess you may want to debug/check other things as well.

Pls send me the updated notebook as well (I want to learn :) )

In case you are too busy, you can copy a sample code how to show the logp here, and I will then update the notebook.

Also, it is no rush mate.

[bwengals]

Yup I'll look at this

[ricardoV94]

I removed the defects label as it was not yet shown that the new results are actually worse.

[ricardoV94]

The incorrect results are probably explained by the bug found in #5923. I'll leave this issue open as we should confirm the results make sense again once that issue is closed.

[danhphan]

Yes, agree. I will test it again once that issue is fixed. Thanks @ricardoV94

[ricardoV94]

Yes, agree. I will test it again once that issue is fixed. Thanks @ricardoV94

Issue is fixed!

[danhphan]

Hi, just re-run these two above examples. It is good for the first example, however, there is still some differences in the second example.

I would say that Find_MAP is improving a lot now. But let's keep this issue for some more days. I will re-run other examples in GP modules sometime at next weekend to confirm. Thanks..

Example 1: The GP salmon notebook

PyMC v3

{'ρ_log__': array(-7.95467461),
 'η_log__': array(-31.66553291),
 'σ_log__': array(3.96207137),
 'ρ': array(0.00035102),
 'η': array(1.76943166e-14),
 'σ': array(52.56609719)}

PyMC v4: This result looks good, similar to PyMC v3.

{'ρ_log__': array(-6.6701119),
 'η_log__': array(-6.11799963),
 'σ_log__': array(3.96206192),
 'ρ': array(0.00126826),
 'η': array(0.00220286),
 'σ': array(52.56560023)}

Example 2: GP-Kron

PyMC v3

{'ls1_log__': array(0.3382134),
 'ls2_log__': array(0.00040654),
 'eta_log__': array(0.9907769),
 'sigma_log__': array(-1.39195968),
 'ls1': array(1.40243975),
 'ls2': array(1.00040662),
 'eta': array(2.69332612),
 'sigma': array(0.24858767)}

PyMC v4: There are little different in ls1 (1.4 vs. 1.0) and eta (2.6 vs. 1.2).

{'ls1_log__': array(0.00615668),
 'ls2_log__': array(0.0012689),
 'eta_log__': array(0.25788484),
 'sigma_log__': array(-1.39516981),
 'ls1': array(1.00617567),
 'ls2': array(1.00126971),
 'eta': array(1.29418977),
 'sigma': array(0.24779095)}

[ricardoV94]

@quantheory would you like to have a look at this?

[quantheory]

I can take a look later this week, but it might be a while before I have time.