ENH: support for nanosecond time in offset and period

[wuan]

This is my attempt to fix the so far missing nanosecond support for pd.DatetimeIndex (see issue #1812).

The Python modification should be relatively easy to review, but unfortunately I had to do a bigger refactoring of period.c in order to support subsecond times.

Im still working on adding more tests and I will try to include the tests on C level as well (test_period.c, which can only be used standalone so far).

I'm looking forward to your remarks and I will be happy to modify the code to fit it into the project.

Cheers
Andreas

[wesm]

@cpcloud this is tragic guys. Major conflict with #2670 i think. can you guys have a look and let me know if it's possible to merge these pull requests without major sadness?

[cpcloud]

I get a build error with the nanosecond_time branch. This is probably because I'm building with bleeding-edge numpy, so I'll try it with the pip default and see if that works.

[ghost]

What's the reason for requiring numpy >=1.7?

[cpcloud]

And why can't I build with bleeding edge numpy? The old_defines.h header should not be included with git master numpy. I'll check this out.

[wuan]

merging #2670 before should be possible. The most collisions are to be expected for period.c but they should be manageable. Only the "int64_t as i8" import introduces a lot of changes which may lead to conflicts.

[wuan]

The reason for requiring numpy >= 1.7 is the support for nanoseconds in np.datetime64 which was not there before 1.7

[wuan]

Maybe its a good idea to remove format only changes in both PRs to ease merging.

[wuan]

Branch did not build with bleeding edge numpy because of
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
in period.h. It should work now.

[ghost]

Will this fail gracefully (or can be made to) if a user has numpy 1.6 installed?
Making numpy>=1.7 a hard dep might be a problem. I believe 1.6 is still, by far,
the common case.

[wuan]

You're right, for up to ms resolution numpy 1.6 should be enough. I'm not shure what it does at the moment, but I will try to make it as compatible as possible.

[ghost]

Great, that would make it easier to get in.

I see you merged upstream into your branch (rather then rebasing), cpcloud can attest to
the pain and suffering this can create.
Would you like some git-fu help to squash things down and rebase on top of master?

[wuan]

Hope it is still possible to switch to rebasing. You are welcome!

[ghost]

@wesm, is #2670 solid enough to use as base for this PR? If you could clear that in, I'll try
and help wuan through git-misery.

[ghost]

@wuan, The last upstream merge you made was against a8abf1f.

try this:

git checkout nanosecond_time   
git checkout -b nanosecond_time_rebase # create identical branch and move to it
git reset a8abf1fdcc3d95306 # point HEAD at latest upstream merged

at this point, you're in the same state as if you were to start with a checkout of a8abf1f, and
then manually made all the changes to the files, without commiting anything.

$ git status
# On branch nanosecond_time_rebase
# Changes not staged for commit:
#   modified:   ci/install.sh
#   modified:   pandas/src/numpy_helper.h
#   modified:   pandas/src/period.c
#   modified:   pandas/src/period.h
#   modified:   pandas/src/util.pxd
#   modified:   pandas/tseries/converter.py
#   modified:   pandas/tseries/frequencies.py
#   modified:   pandas/tseries/offsets.py
#   modified:   pandas/tseries/period.py
#   modified:   pandas/tseries/tests/test_frequencies.py
#   modified:   pandas/tseries/tests/test_offsets.py
#   modified:   pandas/tseries/tests/test_period.py
#   modified:   pandas/tseries/tests/test_timeseries.py
#   modified:   pandas/tseries/tools.py
#   modified:   pandas/tslib.pyx
#   modified:   setup.py
#   modified:   test.sh
#
# Untracked files:

Now you need to break these up into individual commits as best you can.
code, tests, travis and setup can all go into seperate commits.

Of course, you can break things up even further - it's your code, you know best what makes sense.

git add --patch [optional filename]

will let you interactively choose chunks to add to the index, if you want to go that route.

Once that's done, you'll have a neat set of patches based on a8abf1f which
you can rebase to current master.

Following that, mitigating the 1.7 hard dependency is important, and it'll be much easier
to resolve conflicts against #2670.

[ghost]

pushed to 0.12, still need to resolve numpy 1.7 dep issue .

[wuan]

Sorry, my changes were absent during your merge as I was just working on the transition to rebase.

[wuan]

Should I open a new pull request referencing this one, or is it possible to reopen the request?

My branch is now rebased against the pydata master.

Nanoseconds will not work in numpy < 1.7, so some test cases should be skipped in that case. This is still work in progress.

[wesm]

Wait did I miss something was this ready to merge?

[wuan]

As far as I remember we wanted to merge #2670 before. And at least the numpy 1.6 compatibility should be added.

[wesm]

@jreback don't make me take away your push rights =P

[jreback]

shoot, sorry...I didn't mean to merge this one....similary named to another by stephenwlin...

[wesm]

Can someone let me know if this is salvagable?

[jreback]

unless I am reading somewrong, nothing was actually merged here?

[wuan]

most of the changes were not there when the merge was done as I was transforming using rebase. I will have a closer look.