Skip to content

Exception on df.groupby().rolling().agg([f1, f2]) #15409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ohadle opened this issue Feb 15, 2017 · 2 comments
Closed

Exception on df.groupby().rolling().agg([f1, f2]) #15409

ohadle opened this issue Feb 15, 2017 · 2 comments
Labels
Duplicate Report Duplicate issue or pull request Groupby Reshaping Concat, Merge/Join, Stack/Unstack, Explode

Comments

@ohadle
Copy link

ohadle commented Feb 15, 2017

Trying to reproduce example [_85] from the docs:

from pandas import DataFrame, Timestamp

c = pandas.DataFrame({u'ul_payload': {('a', Timestamp('2016-11-01 06:15:00')): 5, ('a', Timestamp('2016-11-01 07:45:00')): 8, ('a', Timestamp('2016-11-01 09:00:00')): 9, ('a', Timestamp('2016-11-01 07:15:00')): 6, ('a', Timestamp('2016-11-01 07:30:00')): 7, ('a', Timestamp('2016-11-01 06:00:00')): 4}, u'dl_payload': {('a', Timestamp('2016-11-01 06:15:00')): 15, ('a', Timestamp('2016-11-01 07:45:00')): 18, ('a', Timestamp('2016-11-01 09:00:00')): 19, ('a', Timestamp('2016-11-01 07:15:00')): 16, ('a', Timestamp('2016-11-01 07:30:00')): 17, ('a', Timestamp('2016-11-01 06:00:00')): 14}})

In [27]: c
Out[27]:
                       dl_payload  ul_payload
a 2016-11-01 06:00:00          14           4
  2016-11-01 06:15:00          15           5
  2016-11-01 07:15:00          16           6
  2016-11-01 07:30:00          17           7
  2016-11-01 07:45:00          18           8
  2016-11-01 09:00:00          19           9

In [30]: c.groupby(level=0).rolling(window=3).agg([np.sum, np.max])
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/olevinkr/repos/research/src/son/research/apps/deep_forecast/data.py in <module>()
----> 1 c.groupby(level=0).rolling(window=3).agg([np.sum, np.max])

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
   1063     @Appender(SelectionMixin._agg_doc)
   1064     def aggregate(self, arg, *args, **kwargs):
-> 1065         return super(Rolling, self).aggregate(arg, *args, **kwargs)
   1066
   1067     agg = aggregate

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
    319
    320     def aggregate(self, arg, *args, **kwargs):
--> 321         result, how = self._aggregate(arg, *args, **kwargs)
    322         if result is None:
    323             return self.apply(arg, args=args, kwargs=kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate(self, arg, *args, **kwargs)
    566             return result, True
    567         elif hasattr(arg, '__iter__'):
--> 568             return self._aggregate_multiple_funcs(arg, _level=_level), None
    569         else:
    570             result = None

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate_multiple_funcs(self, arg, _level)
    611                 try:
    612                     colg = self._gotitem(col, ndim=1, subset=obj[col])
--> 613                     results.append(colg.aggregate(arg))
    614                     keys.append(col)
    615                 except (TypeError, DataError):

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
   1063     @Appender(SelectionMixin._agg_doc)
   1064     def aggregate(self, arg, *args, **kwargs):
-> 1065         return super(Rolling, self).aggregate(arg, *args, **kwargs)
   1066
   1067     agg = aggregate

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
    319
    320     def aggregate(self, arg, *args, **kwargs):
--> 321         result, how = self._aggregate(arg, *args, **kwargs)
    322         if result is None:
    323             return self.apply(arg, args=args, kwargs=kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate(self, arg, *args, **kwargs)
    566             return result, True
    567         elif hasattr(arg, '__iter__'):
--> 568             return self._aggregate_multiple_funcs(arg, _level=_level), None
    569         else:
    570             result = None

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate_multiple_funcs(self, arg, _level)
    595             for a in arg:
    596                 try:
--> 597                     colg = self._gotitem(obj.name, ndim=1, subset=obj)
    598                     results.append(colg.aggregate(a))
    599

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _gotitem(self, key, ndim, subset)
    682                        for attr in self._attributes])
    683         self = self.__class__(subset,
--> 684                               groupby=self._groupby[key],
    685                               parent=self,
    686                               **kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in __getitem__(self, key)
    328     def __getitem__(self, key):
    329         if self._selection is not None:
--> 330             raise Exception('Column(s) %s already selected' % self._selection)
    331
    332         if isinstance(key, (list, tuple, ABCSeries, ABCIndexClass,

Exception: Column(s) dl_payload already selected

Aggregating by a single function doesn't throw the exception (xref #14013):

In [29]: c.groupby(level=0).rolling(window=3).agg(np.sum)
Out[29]:
                         dl_payload  ul_payload
a a 2016-11-01 06:00:00         NaN         NaN
    2016-11-01 06:15:00         NaN         NaN
    2016-11-01 07:15:00        45.0        15.0
    2016-11-01 07:30:00        48.0        18.0
    2016-11-01 07:45:00        51.0        21.0
    2016-11-01 09:00:00        54.0        24.0
INSTALLED VERSIONS ------------------ commit: None python: 2.7.13.final.0 python-bits: 64 OS: Darwin OS-release: 16.4.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 28.6.1
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
statsmodels: 0.6.1
xarray: 0.9.1
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.2
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.9.4
boto: 2.45.0
pandas_datareader: None
@ohadle
Copy link
Author

ohadle commented Feb 15, 2017

Similarly, example [_88]:

In [39]: c.groupby(level=0).rolling(window=3).agg({'dl_payload': [np.sum, np.max], 'ul_payload': [np.sum, np.max]})
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/olevinkr/repos/research/src/son/research/apps/deep_forecast/data.py in <module>()
----> 1 c.groupby(level=0).rolling(window=3).agg({'dl_payload': [np.sum, np.max], 'ul_payload': [np.sum, np.max]})

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
   1063     @Appender(SelectionMixin._agg_doc)
   1064     def aggregate(self, arg, *args, **kwargs):
-> 1065         return super(Rolling, self).aggregate(arg, *args, **kwargs)
   1066
   1067     agg = aggregate

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
    319
    320     def aggregate(self, arg, *args, **kwargs):
--> 321         result, how = self._aggregate(arg, *args, **kwargs)
    322         if result is None:
    323             return self.apply(arg, args=args, kwargs=kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate(self, arg, *args, **kwargs)
    547
    548                 try:
--> 549                     result = _agg(arg, _agg_1dim)
    550                 except SpecificationError:
    551

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _agg(arg, func)
    498                 result = compat.OrderedDict()
    499                 for fname, agg_how in compat.iteritems(arg):
--> 500                     result[fname] = func(fname, agg_how)
    501                 return result
    502

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _agg_1dim(name, how, subset)
    481                     raise SpecificationError("nested dictionary is ambiguous "
    482                                              "in aggregation")
--> 483                 return colg.aggregate(how, _level=(_level or 0) + 1)
    484
    485             def _agg_2dim(name, how):

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
   1063     @Appender(SelectionMixin._agg_doc)
   1064     def aggregate(self, arg, *args, **kwargs):
-> 1065         return super(Rolling, self).aggregate(arg, *args, **kwargs)
   1066
   1067     agg = aggregate

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/window.pyc in aggregate(self, arg, *args, **kwargs)
    319
    320     def aggregate(self, arg, *args, **kwargs):
--> 321         result, how = self._aggregate(arg, *args, **kwargs)
    322         if result is None:
    323             return self.apply(arg, args=args, kwargs=kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate(self, arg, *args, **kwargs)
    566             return result, True
    567         elif hasattr(arg, '__iter__'):
--> 568             return self._aggregate_multiple_funcs(arg, _level=_level), None
    569         else:
    570             result = None

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _aggregate_multiple_funcs(self, arg, _level)
    595             for a in arg:
    596                 try:
--> 597                     colg = self._gotitem(obj.name, ndim=1, subset=obj)
    598                     results.append(colg.aggregate(a))
    599

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in _gotitem(self, key, ndim, subset)
    682                        for attr in self._attributes])
    683         self = self.__class__(subset,
--> 684                               groupby=self._groupby[key],
    685                               parent=self,
    686                               **kwargs)

/Users/olevinkr/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in __getitem__(self, key)
    328     def __getitem__(self, key):
    329         if self._selection is not None:
--> 330             raise Exception('Column(s) %s already selected' % self._selection)
    331
    332         if isinstance(key, (list, tuple, ABCSeries, ABCIndexClass,

Exception: Column(s) ul_payload already selected

@jreback
Copy link
Contributor

jreback commented Feb 15, 2017

this is a dupe of #15072

pull-requests to fix are welcome!

@jreback jreback closed this as completed Feb 15, 2017
@jreback jreback added Duplicate Report Duplicate issue or pull request Groupby Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Feb 15, 2017
@jreback jreback added this to the No action milestone Feb 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Duplicate Report Duplicate issue or pull request Groupby Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jreback @ohadle