DOC: freeze old whatsnew notes?

[jorisvandenbossche]

A problem with using the ipython directory for the whatsnew/release parts of the docs (so that all code examples are run during the doc build), is that after a while when things change/get deprecated, these pages don't build correctly anymore, generating warnings/exceptions.

An example: http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#hdfstore, with the unique attribute that was removed recently in #6813

I don't think we should start updating the older whatsnew docs (they are a kind of historical overview), but we should do something as those doc build warnings are annoying. A proposal:

• after a release (or after some time), we run whatsnew/release once with the pandas version of that release, and replace all ipython directory blocks to literal code-blocks with output.
• we maybe add a warning at the top of each whatsnew that the code examples can be outdated

An alternative is to add :okexcept: to those ipython code-blocks; this would solve the warnings during the doc build, but not the 'faulty' output in the docs itself.

[jreback]

i think maybe easiest to simply comment out the code and say its been replaced in that version (and show the alternative). for most of these this should be easy just an argument name change. I'll put the example up for HDFStore.unique

replacing with code-blocks is ok too (though actually a nice comment IMHO is more informative)

[jorisvandenbossche]

That's also an option.
If we would choose to convert the code examples to code-blocks, this can easily be done by outputting rst instead of html (rst with processed extensions) with something like https://pythonhosted.org/sphinxcontrib-restbuilder/ I think.

But it's indeed maybe simpler and more informative to just adapt and comment that example.
An advantage of converting it to code-blocks would be that the doc builds would be slightly faster.

[jreback]

fixed HDFStore.unique issue in c1440e9

[jreback]

http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#v0-10-1-january-22-2013 (except for the typo of course!)

[jreback]

can you post what's still outstadning on this? (if you have handy)

[jorisvandenbossche]

Not that I know of. There are in any case no doc build warnings/errors at the moment. (but the problem is that you don't always notice it if something has changed in the output of the code so the explanation does not really make sense anymore).

[jreback]

ok...will leave open and i'll guess will just have to review all whats new for sensibility

[jorisvandenbossche]

I think we should just make a choice:

• just fix it when an issue comes up (at least those that we notice due to doc build errors, eg a removed keyword), and so close this issue now (no reason to keep it open I think. Errors will always be noticed in doc build).
• or convert older whatsnews to static code blocks (not too much work I think, should explore it)

The easiest solution is the first, although I have a slight preference for the second (I see it as an historical overview, and not as up to date docs. Downside is that if people are looking at older whatsnews they see possibly outdated code). But no strong opinion on this.

[jreback]

@jorisvandenbossche IIRC you fixed this?

[jreback]

update on this.

These older whatsnew are building pretty well with a couple of minor exceptions.

However, it would be nice to changes examples from using ipython -> code-blocks to:

• make this faster
• make less error prone

[jorisvandenbossche]

This would indeed be nice to do (at least for the very old ones, for the relatively recent ones it is still a good way sometimes to capture errors)

I at some point looked into a kind of 'text' sphinx builder that would output what we want, but never really figured it out. I think it should be somehow possible (eg numpydoc also emits the intermediate generated source code that is never saved in a file to build the actual docstring page when using very verbose mode)

[avinashpancham]

While working on #36838 I got exactly this issue. If I understand it correctly we want to remove the ipython directive in old whatsnew notes and replace it with code-blocks. If we still want to do that, then I can work on this PR

[jorisvandenbossche]

@avinashpancham sorry for the late reply, but yes, I think that's still something we want to do. (ideally we would have a way to do this automatically, though, I think. In theory it should be possible with some sphinx / ipython directive hacks I think)

[fangchenli]

See #41464 for how to freeze old whatsnew notes. It's very simple but labor-intensive.

[jorisvandenbossche]

Yeah, if we want to do this for the later (and much bigger) whatsnew files as well, I think some automatic method (using sphinx / ipython directive) would be better

[derrick-anderson]

I'm digging into older "easy" tickets and hoping to brush off my contribution skills.

I see two issues here:

1. Freeze "old" whatsnew notes in order to prevent syntax errors.
   a.What do we consider old? Anything not the most recent?
2. Find a way to programmatically do this.

For 1. it's definitely labor intensive but not undoable, I'm happy to help tackle.
For 2 however this one scratches an itch i've had for a while TOOLING. I'll start to investigate some patterns. I like the idea of potentially overriding a sphynx directive.

Would we want to split this into two issues? I can submit a new one for the tooling enhancement. Then we could use this one to run a check list of the outstanding docs to convert so that this can still be closed with a goal in mind.

[derrick-anderson]

Hi Everyone!

I have been doing a bunch of digging on this topic. I've come down on this solution that allows you to lock-in a specific version of a library for build time. This would require the builder to install all the possible versions of pandas on the build machine. That is possible with a requires file and then we would include a global import for that specific version at the top of the page for that version. There's also an option to drop that work into the execlines for that page, but then a whole new Sphinx shell would need to be spun up for each page, probably making those builds even longer.

However, over time I am agreeing with the idea that there likely is no real value in re-running all these old code snippets during build time. The only real value I can see is when a change is back-ported then the value in a static code block won't change. This however is a rare occurrence and the docs being wrong should be significantly less of an issue than trying to wedge something into the build machine and putting heavier onus on contributors.

I'm going to get started converting all the old iPython directive blocks into code-block::ipython and drop in the appropriate output from that version of pandas. We should probably note somewhere to use that directive for docs after that.

Let me know if you have other ideas!

-Derrick

[TheMellyBee]

Can this be closed?

[jorisvandenbossche]

I think this would still be nice to do

[TheMellyBee]

Sounds good! This looks like it needs some refactoring, updating, and just basic maintenance. Would you like some help on it?