Update notebook examples with new variational API

[ferrine]

Moving to new variational API, linked with #1968

• lda-advi-aevb.ipynb
• GLM-hierarchical-advi-minibatch.ipynb
• GLM-hierarchical-ADVI.ipynb
• gaussian-mixture-model-advi.ipynb
• convolutional_vae_keras_advi.ipynb

[ferrine]

I found that normalizing loss can make convergence slower due to weaker gradients. I'll make it optional for user in a separate PR

[ferrine]

Previous callback stopped earlier for hierarchical model when convergence was not achieved. The new one did not stop at all though(remark: it stopped once).

[ferrine]

That's done. CC @taku-y, @twiecki

I've changed callback here. This change should be approved or disapproved. In case it's ok I'll revert that commit and change the callback in separate PR

[fonnesbeck]

Can you replace the fit method calls with function calls in the notebooks?

[twiecki]

Agreed. Moreover, the partial construct everywhere is not nice API, can't we pass an object, or optimizer_kwargs?

[ferrine]

@fonnesbeck pm.fit does not return elbo history. In case we want to access inference history we need inference object.

[twiecki]

@ferrine Couldn't we store that in the returned object, like we did before?

[ferrine]

@twiecki good point, but I don't think that's bad api. Since 2 optimizers can be passed in the future, there will be 2 more kwarg

[twiecki]

@ferrine Not bad, but not user-friendly I would argue. Can we not pass in Optimizer(lrate=.5), must it be a class?

[ferrine]

@twiecki ofc we can but ideologically they'd be better distinguished

[twiecki]

@ferrine not sure I follow, distinguished to what?

[fonnesbeck]

I've said this before, but I feel pretty strongly about keeping the default interface consistent across fitting methods -- particularly in our examples.

[ferrine]

I think that pm.fit can return loss history additionally as a separate array object. I do not want to attach new attribute to approximation as loss history is not related with approximation itself.

approx, losses = pm.fit(...)

What about optimizers. What about wrapping them in a classes?
Capital case Adam will represent a class that can be inited before call and lower case adam will remain functions as in Lasagne. So we will just add some lines below every definition

[twiecki]

@ferrine Why wouldn't the loss be part of the approximation? To me it makes sense.

I see now, I somehow thought we already used classes in updates.py, which is clearly wrong. So yes, turning optimizers into classes is the right approach here.

[ferrine]

@twiecki Approximation purpose is to store parameters and allow sampling. Inference purpose is to fit the approx. These two tasks differ a lot. The right design is to make objects serve one particular thing. Mixing inference with approximation violates this idea.

[twiecki]

@ferrine I think the loss history of an approximation can be coherently included in the approximation class. I do see your point, but the API trumps code purity in my opinion. I love the pm.fit() analog to pm.sample(), but having one function return two things and the other just one is incoherent.

[ferrine]

Can I introduce a lightweight wrapper over results so that they store both approximation and stats?

[twiecki]

@ferrine Definitely, how would you do that on a high level?

[ferrine]

I'll try. High level is needed. I always write some weird callbacks to have custom watch list of statistics. Adding watch_list as kwarg to fit will solve the problem and they will be returned with inference result. Later it will be possible to use step methods along with it.

[twiecki]

Sounds good. FWIW callbacks are usually frowned upon, look for callback hell.

[ferrine]

Rebased

[ferrine]

@twiecki following our discussion about VI stats, check out #2230

[ferrine]

Solved by #2171