(WIP) ENH: Add PeriodBlock #13755

sinhrks · 2016-07-22T11:36:07Z

closes API/ENH: allow Period to be a first-class type via a PeriodBlockinternal type #7964
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

This PR is NOT ready for review, but to show basic direction. Especially, supporting period arithmetic is quite unreliable because current _TimeOp class is basically for np.ndarray operations. I think refactoring _TimeOp is needed prior to this PR.

Would like to split the PR to 3 parts:

1. Add PeriodDtype definition, let PeriodIndex to have PeriodDtype like period[M]. (ENH: PeriodIndex now has period dtype #13941)
1. Refactor _TimeOp splitting to _DatetimeOp and _TimedeltaOp, merging DatetimeIndex and TimedeltaIndex arithmetics.
- _DatetimeOp is for lhs or rhs is datetime64 or datetimetz dtypes.
- _TimedeltaOp is for other timedelta related types.
1. Add PeriodBlock, adding _PeriodOp for arithmetic (this PR).

codecov-io · 2016-07-22T12:03:10Z

Current coverage is 85.51% (diff: 84.37%)

Merging #13755 into master will decrease coverage by 0.03%

@@             master     #13755   diff @@
==========================================
  Files           145        145          
  Lines         51404      51655   +251   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43975      44172   +197   
- Misses         7429       7483    +54   
  Partials          0          0

Powered by Codecov. Last update 8e13da2...80e9caf

jreback · 2016-07-22T13:48:07Z

pandas/core/internals.py

@@ -2451,6 +2462,182 @@ def shift(self, periods, axis=0, mgr=None):
                                           placement=self.mgr_locs)]


+class PeriodBlock(NonConsolidatableMixIn, DatetimeBlock):


you might be better actoring out some of the holding Index suff from DatetimeTZBLock and creating IndexHolderBlock (or better name). then sub-classing for DatetimeTZBlock and PeriodBlock. Should be in theory less duplication.

jreback · 2016-08-03T10:38:33Z

pandas/core/internals.py

+        # delegate
+        return super(PeriodBlock, self)._astype(dtype=dtype, **kwargs)
+
+    def external_values(self):


can move to IndexHolder

jreback · 2016-08-07T14:52:17Z

pandas/core/reshape.py

        if values.ndim == 1:
            if isinstance(values, Categorical):
                self.is_categorical = values
                values = np.array(values)
+            elif is_period(values):
+                self.is_period = values
+                values = values.values


I guess not handling other i8 here ATM? (e.g. datetime/datetimetz)?

jreback · 2016-08-07T15:00:57Z

@sinhrks looks pretty good. certainly can merge mostly as is and just make a to-do on other issues. Looks like pretty much works and doesn't change the api much.

Just on a small whatsnew section to we can keep updating.

jorisvandenbossche · 2016-08-31T10:50:38Z

@sinhrks @jreback What is the status of this one? Still try to include it in 0.19.0 or push to 0.20?

jreback · 2016-08-31T13:33:24Z

ideally I'd like this to go in. I think most of the prep work has been done.

jorisvandenbossche · 2016-11-24T21:36:44Z

@sinhrks Is this ready for review? (no docs yet, and some todo's listed in the code) Just to prioritize my time :-)

sinhrks · 2016-11-24T23:39:48Z

Not yet. I'll make a notice once ready, hopefully this weekend... :)

jreback · 2017-01-19T02:08:25Z

pandas/core/reshape.py

@@ -69,10 +69,14 @@ def __init__(self, values, index, level=-1, value_columns=None,
                 fill_value=None):

        self.is_categorical = None
+        self.is_period = None
        if values.ndim == 1:
            if isinstance(values, Categorical):


-> is_categorical

jreback · 2017-01-19T02:10:11Z

pandas/io/tests/test_feather.py

@@ -84,7 +84,7 @@ def test_unsupported(self):

        # period
        df = pd.DataFrame({'a': pd.period_range('2013', freq='M', periods=3)})
-        self.check_error_on_write(df, ValueError)
+        self.check_error_on_write(df, feather.FeatherError)


yeah I don't think feather supports Periods ATM.

jreback · 2017-01-19T02:11:05Z

pandas/src/hashtable_class_helper.pxi.in

@@ -488,6 +506,8 @@ cdef class StringHashTable(HashTable):
            if k != self.table.n_buckets:
                val = self.table.vals[k]

+    @cython.wraparound(False)


this file changes can do as a separate PR?

jreback · 2017-01-19T02:12:45Z

pandas/tseries/frequencies.py

@@ -773,7 +772,7 @@ def infer_freq(index, warn=True):
                            "dtype on a Series of {0}".format(index.dtype))
        index = values

-    if is_period_arraylike(index):
+    if isinstance(index, pd.PeriodIndex):


is_period_dtype

jreback · 2017-01-19T02:13:29Z

pandas/types/cast.py

+        from pandas.tseries.period import Period
+        if isnull(fill_value):
+            fill_value = tslib.iNaT
+        elif isinstance(fill_value, Period):


you can use ABCPeriod here (or is_period) maybe better

jreback · 2017-01-19T02:15:16Z

pandas/types/common.py

@@ -73,7 +73,7 @@ def is_datetimetz(array):

 def is_period(array):
    """ return if we are a period array """
-    return isinstance(array, ABCPeriodIndex) or is_period_arraylike(array)


hmm so this is now the same as is_period_dtype, maybe we should just deprecate this then, unless this is really equiv to isinstance(val, Period) ? IOW a scalar

jreback · 2017-05-07T14:00:15Z

let's close for now. If you want to rebase and fixup lmk. Would like to do this (maybe in concert with a IntervalBlock).

sinhrks added Enhancement Performance Memory or execution speed performance Period Period data type labels Jul 22, 2016

sinhrks added this to the 0.19.0 milestone Jul 22, 2016

jreback reviewed Jul 22, 2016
View reviewed changes

sinhrks force-pushed the periodblock6 branch from 028f98e to 858b843 Compare July 23, 2016 08:05

sinhrks force-pushed the periodblock6 branch 4 times, most recently from dc92e4a to 8b5ea28 Compare August 2, 2016 11:51

jreback reviewed Aug 3, 2016
View reviewed changes

sinhrks force-pushed the periodblock6 branch 2 times, most recently from bcfb433 to 5596c3d Compare August 7, 2016 12:50

jreback reviewed Aug 7, 2016
View reviewed changes

sinhrks force-pushed the periodblock6 branch 2 times, most recently from 661e4d2 to 8d8ef2f Compare August 17, 2016 22:50

jankatins mentioned this pull request Aug 18, 2016

Require a problem description in PRs #14031

Closed

jorisvandenbossche modified the milestones: 0.19.0rc, 0.19.0 Sep 1, 2016

jreback modified the milestones: 0.20.0, 0.19.0rc Sep 5, 2016

sinhrks force-pushed the periodblock6 branch from 8d8ef2f to 321e337 Compare October 30, 2016 13:27

sinhrks force-pushed the periodblock6 branch from 321e337 to 3cdb115 Compare November 20, 2016 01:06

sinhrks force-pushed the periodblock6 branch from 0ac8737 to e57c684 Compare November 29, 2016 21:52

sinhrks force-pushed the periodblock6 branch from e57c684 to 371d491 Compare January 8, 2017 05:59

sinhrks added 4 commits January 19, 2017 08:07

ENH: Add PeriodBlock

bd07aab

Fix lint errors

d9a5eb8

Remove unnecessary path

0f2b03a

Additional fixes

80e9caf

sinhrks force-pushed the periodblock6 branch from dfea6da to 80e9caf Compare January 18, 2017 23:26

jreback reviewed Jan 19, 2017

View reviewed changes

jreback removed this from the 0.20.0 milestone Mar 23, 2017

jreback closed this May 7, 2017

jorisvandenbossche added the Closed PR label May 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(WIP) ENH: Add PeriodBlock #13755

(WIP) ENH: Add PeriodBlock #13755

sinhrks commented Jul 22, 2016 •

edited

Loading

codecov-io commented Jul 22, 2016 •

edited

Loading

jreback Jul 22, 2016

jreback Aug 3, 2016

jreback Aug 7, 2016

jreback commented Aug 7, 2016

jorisvandenbossche commented Aug 31, 2016

jreback commented Aug 31, 2016

jorisvandenbossche commented Nov 24, 2016

sinhrks commented Nov 24, 2016

jreback Jan 19, 2017

jreback Jan 19, 2017

jreback Jan 19, 2017

jreback Jan 19, 2017

jreback Jan 19, 2017 •

edited

Loading

jreback Jan 19, 2017

jreback commented May 7, 2017

		@@ -2451,6 +2462,182 @@ def shift(self, periods, axis=0, mgr=None):
		placement=self.mgr_locs)]


		class PeriodBlock(NonConsolidatableMixIn, DatetimeBlock):

(WIP) ENH: Add PeriodBlock #13755

(WIP) ENH: Add PeriodBlock #13755

Conversation

sinhrks commented Jul 22, 2016 • edited Loading

codecov-io commented Jul 22, 2016 • edited Loading

Current coverage is 85.51% (diff: 84.37%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 7, 2016

jorisvandenbossche commented Aug 31, 2016

jreback commented Aug 31, 2016

jorisvandenbossche commented Nov 24, 2016

sinhrks commented Nov 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Jan 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented May 7, 2017

sinhrks commented Jul 22, 2016 •

edited

Loading

codecov-io commented Jul 22, 2016 •

edited

Loading

jreback Jan 19, 2017 •

edited

Loading