ENH: to_csv() date formatting

[qwhelan]

This commit adds support for formatting datetime object output from to_csv()
closes #2583

In [3]: spx = DataReader('^GSPC', data_source='yahoo')

In [4]: spx.head()
Out[4]: 
               Open     High      Low    Close      Volume  Adj Close
Date                                                                 
2010-01-04  1116.56  1133.87  1116.56  1132.99  3991400000    1132.99
2010-01-05  1132.66  1136.63  1129.66  1136.52  2491020000    1136.52
2010-01-06  1135.71  1139.19  1133.95  1137.14  4972660000    1137.14
2010-01-07  1136.27  1142.46  1131.32  1141.69  5270680000    1141.69
2010-01-08  1140.52  1145.39  1136.22  1144.98  4389590000    1144.98

In [5]: spx.to_csv('spx_temp.csv', date_format='%Y%m%d')

In [6]: !head spx_temp.csv
Date,Open,High,Low,Close,Volume,Adj Close
20100104,1116.56,1133.87,1116.56,1132.99,3991400000,1132.99
20100105,1132.66,1136.63,1129.66,1136.52,2491020000,1136.52
20100106,1135.71,1139.19,1133.95,1137.14,4972660000,1137.14
20100107,1136.27,1142.46,1131.32,1141.69,5270680000,1141.69
20100108,1140.52,1145.39,1136.22,1144.98,4389590000,1144.98
20100111,1145.96,1149.74,1142.02,1146.98,4255780000,1146.98
20100112,1143.81,1143.81,1131.77,1136.22,4716160000,1136.22
20100113,1137.31,1148.4,1133.18,1145.68,4170360000,1145.68
20100114,1145.68,1150.41,1143.8,1148.46,3915200000,1148.46

The date_format= keyword will be applied to every element of a DatetimeIndex (index or columns) and DatetimeBlock (values). It works for both the Python engine and the new Cython engine:

In [7]: datetimes = DataFrame({spx.index[0]: spx.index}, index=spx.index).head()

In [8]: datetimes
Out[8]: 
                    2010-01-04
Date                          
2010-01-04 2010-01-04 00:00:00
2010-01-05 2010-01-05 00:00:00
2010-01-06 2010-01-06 00:00:00
2010-01-07 2010-01-07 00:00:00
2010-01-08 2010-01-08 00:00:00

In [9]: datetimes.to_csv('datetimes_temp.csv', date_format='%m/%d/%Y')

In [10]: !head datetimes_temp.csv
Date,01/04/2010
01/04/2010,01/04/2010
01/05/2010,01/05/2010
01/06/2010,01/06/2010
01/07/2010,01/07/2010
01/08/2010,01/08/2010

In [11]: datetimes.to_csv('datetimes_temp.csv', date_format='%m/%d/%Y', engine='python')

In [12]: !head datetimes_temp.csvDate,01/04/2010
01/04/2010,01/04/2010
01/05/2010,01/05/2010
01/06/2010,01/06/2010
01/07/2010,01/07/2010
01/08/2010,01/08/2010

Let me know if there are any questions or issues.

[qwhelan]

@jreback @cpcloud Any chance this could make it in before the v0.12 release?

Also, I forgot to mention above that this doesn't handle MultiIndexes. I don't see a clean way to do it, but I can revisit it if someone requests it.

[cpcloud]

I think only bug fixes for now. Really trying to get v0.12 out ASAP!

[qwhelan]

Alright, just wanted to check.

Thanks.

[cpcloud]

Btw Thanks for the Pr!

[jreback]

@qwhelan I think you need to either raise or warn if the stringified datetimes are not == to the current

otherwise its pretty easy to chop say datetimes to dates (which is fine, except it should be done explicity by resetting the index, rather than a typo/incorrect format)

# should be in core/index.py
def hastimes(self):
    return not (set(self.times) == set([time(0,0)]))

in core/internals/DatetimeBlock
if.date_format is not None:

   # values might be a series here (e.g. a column from a frame)
   # need to convert to an index to test this
   if values.hastimes():

        # test converted == values, maybe by sampling
        # or can do:   (values == pd.to_datetime(converted)).all()

        # if not == raise/warn

[qwhelan]

@jreback I don't see hastimes() or anything with the same functionality. I'll add unless there's another function I should be using (and probably put it in tseries/tindex.py).

[jreback]

yep have to add it (that's why I put it out there!)

[jreback]

@qwhelan can you rebase to current?

this looks pretty good otherwise

[jreback]

@qwhelan this somehow got lost....can you rebase to current master.....can get this in for 0.13...thxs

[jreback]

also...need to make sure this handles NaT as well (e.g. use in your sample tests)

[jtratner]

perf test?

[qwhelan]

Sorry for neglecting this. I'll make the changes later this week.

[qwhelan]

Added a perf test and tested/handled NaTs.

I was caught in rebase hell for the last week, but it should be ready to go unless there are additional requests.

[jreback]

move example to 0.13.0 (from 0.12)

[jreback]

this changed in master, (as the timedelta stuff was moved); pls start with the current and don't add (which prob happened in a rebase)

[jreback]

@qwhelan almost there....just need those 2 changes!

[qwhelan]

@jreback Just pushed the changes. Let me know if you're looking for something different.

[jreback]

@qwhelan can you move the v0.12.0 announcement to v0.13.0?
change the test that uses NaT to include some dates as well as NaT's

[qwhelan]

@jreback Made those changes.

[jreback]

this can all be collapsed down, something like:

if date_format is None:
        date_formatter = lambda x: x._repr_base
else 
        date_formatter = lambda x: x.strftime(date_format) if notnull(x)

if isinstance(val, (np.datetime64, datetime.datetime)):
       val = date_formatter(lib.Timestamp(val))

[jreback]

@qwhelan can you update to my comments....?

[qwhelan]

@jreback All done. Thanks for the suggestions.

[jreback]

thanks...merged