Skip to content

COMPAT: Further Expand Compatibility with fromnumeric.py #13148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

gfyoung
Copy link
Member

@gfyoung gfyoung commented May 12, 2016

Follow-on to #12810 by expanding compatibility with fromnumeric.py in the following modules:

  1. tslib.pyx
  2. window.py
  3. groupby.py and resample.py (shared classes)

Closes #12811.

@jreback jreback added the Compat pandas objects compatability with Numpy or Python functions label May 12, 2016
@jreback jreback added this to the 0.18.2 milestone May 12, 2016
@@ -70,6 +70,34 @@ New Behavior:

type(s.tolist()[0])

numpy function compatibility
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just make this a 1-liner. This is not something we care to advertise.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. Done.

@gfyoung gfyoung force-pushed the fromnumeric-compat-continued branch from e5465a6 to ca890e9 Compare May 12, 2016 21:26
@codecov-io
Copy link

codecov-io commented May 12, 2016

Current coverage is 84.14%

Merging #13148 into master will decrease coverage by <.01%

@@             master     #13148   diff @@
==========================================
  Files           138        138          
  Lines         50394      50409    +15   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42409      42416     +7   
- Misses         7985       7993     +8   
  Partials          0          0          

Powered by Codecov. Last updated by 2a120cf...e5465a6

@jreback
Copy link
Contributor

jreback commented May 13, 2016

@gfyoung let's maybe just disallow using numpy functions on window/groupby ops (the Timestamp one is prob ok).

this should just raise a nicer error message (say this is not supported)

In [6]: np.mean(Series(range(3)).groupby(range(3)))
TypeError: mean() got an unexpected keyword argument 'axis'

@gfyoung
Copy link
Member Author

gfyoung commented May 13, 2016

Fair enough. So instead of validating as I do now, I just check if the args or kwargs contain any numpy-specific arguments (not sure how else to check if the pandas function is being called from numpy otherwise)?

@gfyoung gfyoung force-pushed the fromnumeric-compat-continued branch 3 times, most recently from 85fa8da to 8cb265d Compare May 14, 2016 02:17
@jreback
Copy link
Contributor

jreback commented May 14, 2016

not really sure how to figure out if called via numpy w/o stack inspection (not a good idea!).

though you could simply just have it raise when it validates, e.g. its sort of duck like in the argument signature when called from numpy. IOW numpy is always passing dtype/out I think. So just raise a nicer error message on that (you prob need to pass a parameter to your validation routines to say hey if this is not validation, then print this nice error message; e.g. you want to say something like)

""" numpy operations are not valid with groupy, use .groupby(...).mean() instead

@gfyoung gfyoung force-pushed the fromnumeric-compat-continued branch 3 times, most recently from 220272e to 2a5817d Compare May 18, 2016 14:31
@@ -25,6 +25,14 @@
from pandas.compat import OrderedDict


class BadNumpyCall(Exception):
Copy link
Contributor

@jreback jreback May 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm not sure I like the name. maybe UnsupportedFunctionCall? and move to core.common. inherit from ValueError

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works too. Will change.

@gfyoung gfyoung force-pushed the fromnumeric-compat-continued branch 2 times, most recently from 659dd51 to e92aa9f Compare May 18, 2016 23:37
@@ -46,7 +46,8 @@ API changes


- Non-convertible dates in an excel date column will be returned without conversion and the column will be ``object`` dtype, rather than raising an exception (:issue:`10001`)

- Compatibility with ``fromnumeric.py`` in NumPy has been expanded to timestamps (:issue: `12811`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give a easier explanation? Users don't know what fromnumeric.py is

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Expands compatibility with fromnumeric.py in tslib.pyx and
puts checks in window.py, groupby.py, and resample.py to
ensure that pandas functions such as 'mean' are not called
via the numpy library.

Closes pandas-devgh-12811.
@gfyoung gfyoung force-pushed the fromnumeric-compat-continued branch from e92aa9f to eb4762c Compare May 19, 2016 22:07
@gfyoung
Copy link
Member Author

gfyoung commented May 20, 2016

@jreback, @jorisvandenbossche : Made the changes, and Travis is giving the green light. Ready to merge if there is nothing else.

@jreback jreback closed this in fecb2ca May 20, 2016
@jreback
Copy link
Contributor

jreback commented May 20, 2016

thanks @gfyoung

@gfyoung gfyoung deleted the fromnumeric-compat-continued branch May 20, 2016 17:53
@jreback
Copy link
Contributor

jreback commented May 20, 2016

@gfyoung breaking with numpy master on rounding: https://travis-ci.org/pydata/pandas/jobs/131735642

@gfyoung
Copy link
Member Author

gfyoung commented May 20, 2016

Okay, I think we might need to undo this compatibility change to tslib.pyx. If anyone complains about it, say that "round in numpy is for arrays and not timestamps, and you shouldn't be doing that" (currently, the function will throw an error because keyword arguments like decimals and out are not in the signature and shouldn't be). Here is why:

# signature in tslib.pyx
def round(self, freq, *args, **kwargs):
...

# signature in numpy master
def around(a, decimals=0, out=None):
    return _wrapfunc(a, 'round', decimals=decimals, out=out)

When we call np.round(timestamp, freq), freq is being passed in as the decimals argument. However, decimals is being passed in with a **kwargs bundle, so it isn't being picked up as the positional argument needed for freq. I forgot that that change was not in v1.11.0 but was in master. My bad for not checking.

@jreback
Copy link
Contributor

jreback commented May 20, 2016

ok so just revert that part of the PR
compat for timestamps is not s big deal for this

jreback pushed a commit that referenced this pull request May 21, 2016
Title is self-explanatory.  xref #13148.

Author: gfyoung <[email protected]>

Closes #13246 from gfyoung/tslib-compat-undo and squashes the following commits:

66160f9 [gfyoung] Reverse numpy compat changes to tslib.pyx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compat pandas objects compatability with Numpy or Python functions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants