ENH: implement __iter__ for window objects

[jreback]

xref #11603

might be slightly non-trivial as the current impl does NOT use an explict iterator, rather does a sliding window in cython and marginal calcs.

[kawochen]

I don't know if this is too ambitious, but for custom rolling aggregation, if you can let users supply a StateVariables object which is initialized using the first window, and supports two methods, evolve() and compute(), then you may be able to retain the efficiency when the evolution of the state variables is governed by a simple set of equations. __iter__ becomes a special case of this where the StateVariables contains the whole window, and evolve moves the window forward/replaces the old window with the new window, and compute just returns the data structure inside.

[selik]

Currently, iterating over a pandas.core.window.Rolling object will produce bizarre behavior. For DataFrames as it looks for columns named 0, 1, 2, etc. until it hits a KeyError. For Series it raises AttributeError as it looks for a "columns" attribute.

Until the __iter__ is implemented correctly, perhaps there should be a temporary measure of simply making it non-iterable by raising NotImplementedError.

[jreback]

@selik if you want to submit a PR to raise NotImplementedError would be great!

[antoinecomp]

Okay, how to cope with this problem then ?

I have an object, EWM [com=9.5,min_periods=0,adjust=True,ignore_na=False,axis=0] that I want to add to a dataframe: predictions_df_list['ewma'] which raised this issue

[selik]

@antoinecomp Could you share a bit more of your code and the error message? The traceback will help. Also, what kind of object is EWM?

[antoinecomp]

@selik Sorry for answering so late. This is the code I used:

predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

And this is the error message:

NotImplementedError                       Traceback (most recent call last)
<ipython-input-153-2a1b97ea7cf7> in <module>
----> 1 predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
   3117         else:
   3118             # set column
-> 3119             self._set_item(key, value)
   3120 
   3121     def _setitem_slice(self, key, value):

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _set_item(self, key, value)
   3192 
   3193         self._ensure_valid_index(value)
-> 3194         value = self._sanitize_column(key, value)
   3195         NDFrame._set_item(self, key, value)
   3196 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _sanitize_column(self, key, value, broadcast)
   3385             value = _sanitize_index(value, self.index, copy=False)
   3386 
-> 3387         elif isinstance(value, Index) or is_sequence(value):
   3388             from pandas.core.series import _sanitize_index
   3389 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\dtypes\inference.py in is_sequence(obj)
    470 
    471     try:
--> 472         iter(obj)  # Can iterate over it.
    473         len(obj)   # Has a length associated with it.
    474         return not isinstance(obj, string_and_binary_types)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\window.py in __iter__(self)
    184     def __iter__(self):
    185         url = 'https://github.com/pandas-dev/pandas/issues/11704'
--> 186         raise NotImplementedError('See issue #11704 {url}'.format(url=url))
    187 
    188     def _get_index(self, index=None):

NotImplementedError: See issue #11704 https://github.com/pandas-dev/pandas/issues/11704

[selik]

It looks like is_sequence's test of iter(obj) catching TypeError and AttributeError is inconsistent with __iter__ raising NotImplementedError.

pandas/pandas/core/dtypes/inference.py

Line 495 in caf462c

except (TypeError, AttributeError):

Reverting to the old behavior won't fix this, because iter(obj) would cause a KeyError, which is also unexpected.

@topper-123 @evectant I'm not familiar with the ewm method. Is a NotImplementedError reasonable here?

@jreback Should is_sequence catch NotImplementedError in addition to TypeError and AttributeError when trying iter(obj)?

[TexFly]

I'm getting the same error in core\windows.py
raise NotImplementedError("See issue #11704 {url}".format(url=url))

Any solution or idea how to fix or troubleshoot?
Thanks!

[selik]

@TexFly Are you trying to iterate over a window object? If so, it's as the error says: not implemented.

[mjwillson]

It's easy to run into this error if you (forgivably, IMO) assume that series.ewm(...) will return another Series, and you try to iterate over it.

In fact it's returning a pandas.core.window.EWM object, and you have to call .mean() on it to get the actual exponentially weighted moving average.

[yasersakkaf]

@antoinecomp Did you solve the error. I am getting the same.

[TomAugspurger]

There's a stalled branch at #27399 if anyone wants to pick this up.

[fallenreaper]

I have a Dataframe i was planning to use with ewm, but it seems that when I say:

dt["myCol"].ewm(span=21, ignore_na=true)

it will tell me it is a NotImplementedError and points to come to this Github error. Is there a new way we should be doing this, in either 0.25.0 or 1.0.1?

[harimohanshrm7]

@antoinecomp
this error will go away if you use an aggregator function at the end of statement.
e.g.
predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60).mean()

[banderlog]

In which pandas version it will appear?

The pandas-1.04 still shows NotImplementedError

[TomAugspurger]

1.1 (See the "milestone" on this issue).