TypeError: mean() got an unexpected keyword argument 'dtype'

[yarikoptic]

Numpy 1.8 API change in mean?

/home/data/flash_supression/yarik/scripts/utils.pyc in paired_t_anal(d, value, testcol, rows, cols, print_pt, plots, aggfunc)
    162     pvalues = np.array(pvalues, ndmin=1)
    163 
--> 164     pt_paired_t = DataFrame(np.array([np.mean(v0, axis=0),
    165                                       np.mean(v1, axis=0),
    166                                       np.mean(v10, axis=0),

/usr/lib/pymodules/python2.7/numpy/core/fromnumeric.pyc in mean(a, axis, dtype, out, keepdims)
   2481         try:
   2482             mean = a.mean
-> 2483             return mean(axis=axis, dtype=dtype, out=out)
   2484         except AttributeError:
   2485             pass

TypeError: mean() got an unexpected keyword argument 'dtype'

complete "line" calling this is

    pt_paired_t = DataFrame(np.array([np.mean(v0, axis=0),
                                      np.mean(v1, axis=0),
                                      np.mean(v10, axis=0),
                                      tvalues, pvalues]),
                            index=Index(['mean(%s)' % lvalues[0],
                                         'mean(%s)' % lvalues[1],
                                         'mean effect',
                                         't-score', 'p-value'],
                                        name=contrast_s),
                            columns = columns)

it used to work before ... did not check for sure yet though if it is pandas or recent numpy upgrade (if I did any)

more information:

ipdb> print a.mean
<bound method DataFrame.mean of cond                full   profile
subject                           
01jul10sc.dat   1.572131  1.569658
01oct10cs.dat   1.491370  1.678300
...

and if to provide ndarray compatibility here then interface should match numpy's where dtype is a valid argument to mean

[jreback]

the mean call looks to be in the numeric code

[yarikoptic]

damn -- my reply didn't post from email for some reason here it all is:

On Thu, 18 Jul 2013, jreback wrote:

the mean call looks to be in the numeric code

yeap -- but it calls DataFrame.mean... and apparently it always did, but
differently (see well below). Here is all my blurb:

ok -- there seems to be indeed a recent change in numpy in 'mean':

dec4f4b7 numpy/core/fromnumeric.py (Charles Harris      2013-04-26 21:31:12 -0600 2556)     if type(a) is not mu.ndarray:
0fa4f22f numpy/core/fromnumeric.py (Mark Wiebe          2011-08-17 22:03:06 -0700 2557)         try:
0fa4f22f numpy/core/fromnumeric.py (Mark Wiebe          2011-08-17 22:03:06 -0700 2558)             mean = a.mean
0fa4f22f numpy/core/fromnumeric.py (Mark Wiebe          2011-08-17 22:03:06 -0700 2559)             return mean(axis=axis, dtype=dtype, out=out)
0fa4f22f numpy/core/fromnumeric.py (Mark Wiebe          2011-08-17 22:03:06 -0700 2560)         except AttributeError:
0fa4f22f numpy/core/fromnumeric.py (Mark Wiebe          2011-08-17 22:03:06 -0700 2561)             pass

so the first check changed:

@@ -2527,7 +2527,7 @@ def mean(a, axis=None, dtype=None, out=None, keepdims=False):
     0.55000000074505806

     """
-    if not (type(a) is mu.ndarray):
+    if type(a) is not mu.ndarray:
         try:
             mean = a.mean
             return mean(axis=axis, dtype=dtype, out=out)

which changed nothing for us here I believe... so I am a bit confused... but here is a proof that it used to work before ;):

# on my laptop
$> python -c 'from pandas import DataFrame; import numpy as np; print np.mean(DataFrame([1,2,3]))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/pymodules/python2.7/numpy/core/fromnumeric.py", line 2483, in mean
    return mean(axis=axis, dtype=dtype, out=out)                                   
TypeError: mean() got an unexpected keyword argument 'dtype'

# on a wheezy box
$> python -c 'from pandas import DataFrame; import numpy as np; print np.mean(DataFrame([1,2,3]))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/pymodules/python2.7/numpy/core/fromnumeric.py", line 2374, in mean
    return mean(axis, dtype, out)
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 4772, in mean
    numeric_only=None)                                                            
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 4888, in _reduce
    labels = self._get_agg_axis(axis)                                                
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 4978, in _get_agg_axis
    raise Exception('Must have 0<= axis <= 1')
Exception: Must have 0<= axis <= 1

$> python -c 'from pandas import DataFrame; import numpy as np; print np.mean(DataFrame([1,2,3]), axis=0)'
0    2

$> dpkg -l python-{numpy,pandas}
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version            Architecture   Description
+++-================-==================-==============-=======================================================================================================
ii  python-numpy     1:1.6.2-1.2        amd64          Numerical Python adds a fast array facility to the Python language
ii  python-pandas    0.10.1-1~nd70+1    all            data structures for "relational" or "labeled" data

and there such call results in numpy's code:

2370        try:
2371            mean = a.mean
2372        except AttributeError:
2373            return _wrapit(a, 'mean', axis, dtype, out)
2374 ->     return mean(axis, dtype, out)

so -- it called pandas' mean providing None's for dtype and out as positional
arguments, instead of keyword arguments, which worked. now it doesn't

[jreback]

is there a reason you are not calling

DataFrame(....),mean()

or you can directly pass np.mean(DataFrame(...).values)

pandas mean doesn't accept these extra arguments; I believe they are also somewhat new in numpy

a frame can be heterogenous dtype so this is not really the same

[yarikoptic]

well -- I was not doing anything like that simply because it was working as it should have before. And I assumed that numpy would first 'asanyarray' its argument, but it seems to be just falls through for anything which has 'mean' -- which is imho a wrong logic... as such, you are free to close it -- I think I will whine on numpy issues/list about it... alternatively you could implement dtype allowing e.g. with dtypes per column. Then you might save mortals like me some time adjusting our codes ;)

[yarikoptic]

on the second thought -- I think you might still like to maintain compatibility with numpy's mean API. I thought that DataFrames with those methods were "incepted" with idea of easy integration with numpy routines, etc. So it might be logical to maintain such compatibility, especially when it comes at low cost, e.g. here even adding a dummy "dtype=None" default kwarg and then raising TypeError only if non-None dtype is provided, already might come handy at times

[jreback]

you are using numpy 1.8 dev here? I guess that's an API change from them.....

[jreback]

closing in favor of #4435 (same issue)