fix issue #8000 - interpolation extrapolates over trailing missing values

[grahamjeffries]

This pull request is in response to issue #8000.

Changes to core/common.py add np.nan as the default value for missing values to the left and right non-missing values during interpolation. This prevents DataFrame.interpolate() from extrapolating the last non-missing value over all trailing missing values (the default).

Changes to tests/test_generic.py add test coverage to the above change. A passing test is where an interpolated series with a trailing missing value maintains that trailing missing value after interpolation.

[TomAugspurger]

@grahamjeffries see my comment #8000 (comment)

Any interest in doing a fix? I don't think we need to add left and right kwds to .interpolate() like np.interp has. We already have .fillna which is better for these things. I'd say we can add a keyword like extrapolate or extend or something like that to control whether things get filled on.

Right now we can just keep it as extrapolate=True (for compatibility), but warn that it may change in the future and recommend that people set it to false and fillna after interpolating.

I can take this if you aren't able/interested.

[grahamjeffries]

@TomAugspurger, I'd like to try my hand at this fix. It's my first though so I'll ask for your patience and close review. I'll get to this sometime next week, I suspect

[TomAugspurger]

No rush at all. This has been "broken" for at least a year :)

[jreback]

@grahamjeffries want to revist this?

[jreback]

@grahamjeffries can you revisit this?

[grahamjeffries]

I won't be able to get to this for at least a month. If there's someone else willing and able to make the fix, I'd encourage them to do so. Otherwise, I'll make an effort at that point.

[jreback]

closing as stale, but if you would like to reopen and fixup pls do.

[cancan101]

@jreback What is missing from this PR to get it merged? It seems like a bare minimum:

• add extrapolate argument that defaults to False. Potentially warn if the user does not set.

[jreback]

you could add an extrapolate kw, and set it to None for now (which will do nothing ATM, and not warn). Then in 0.19 could do a warning if its not set. So not much really to fix this.