ENH: enhancements to Panel.apply to enable arbitrary functions and multi-dim slicing (GH1148)

[jreback]

closes #1148

A reproduction of the new docs section

Applying with a Panel will pass a Series to the applied function. If the applied function returns a Series, the result of the application will be a Panel. If the applied function reduces to a scalar, the result of the application will be a DataFrame.

Note Prior to 0.13.1 apply on a Panel would only work on ufuncs (e.g. np.sum/np.max).

In [120]: import pandas.util.testing as tm

In [121]: panel = tm.makePanel(5)

In [122]: panel

<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D

In [123]: panel['ItemA']

                   A         B         C         D
2000-01-03  0.166882 -0.597361 -1.200639  0.174260
2000-01-04 -1.759496 -1.514940 -1.872993 -0.581163
2000-01-05  0.901336 -1.640398  0.825210  0.087916
2000-01-06 -0.317478 -1.130643 -0.392715  0.416971
2000-01-07 -0.681335 -0.245890 -1.994150  0.666084
[5 rows x 4 columns]

A transformational apply.

In [124]: result = panel.apply(lambda x: x*2, axis='items')

In [125]: result

<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D

In [126]: result['ItemA']

                   A         B         C         D
2000-01-03  0.333764 -1.194722 -2.401278  0.348520
2000-01-04 -3.518991 -3.029880 -3.745986 -1.162326
2000-01-05  1.802673 -3.280796  1.650421  0.175832
2000-01-06 -0.634955 -2.261286 -0.785430  0.833943
2000-01-07 -1.362670 -0.491779 -3.988300  1.332168
[5 rows x 4 columns]

A reduction operation.

In [127]: panel.apply(lambda x: x.dtype, axis='items')

                  A        B        C        D
2000-01-03  float64  float64  float64  float64
2000-01-04  float64  float64  float64  float64
2000-01-05  float64  float64  float64  float64
2000-01-06  float64  float64  float64  float64
2000-01-07  float64  float64  float64  float64
[5 rows x 4 columns]

A similar reduction type operation

In [128]: panel.apply(lambda x: x.sum(), axis='major_axis')

      ItemA     ItemB     ItemC
A -1.690090  1.840259  0.010754
B -5.129232  0.860182  0.178018
C -4.635286  0.545328  2.456520
D  0.764068 -3.623586  1.761541
[4 rows x 3 columns]

This last reduction is equivalent to

In [129]: panel.sum('major_axis')

      ItemA     ItemB     ItemC
A -1.690090  1.840259  0.010754
B -5.129232  0.860182  0.178018
C -4.635286  0.545328  2.456520
D  0.764068 -3.623586  1.761541
[4 rows x 3 columns]

A transformation operation that returns a Panel, but is computing the z-score across the major_axis.

In [130]: result = panel.apply(lambda x: (x-x.mean())/x.std(), axis='major_axis')

In [131]: result

<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D

In [132]: result['ItemA']

                   A         B         C         D
2000-01-03  0.509389  0.719204 -0.234072  0.045812
2000-01-04 -1.434116 -0.820934 -0.809328 -1.567858
2000-01-05  1.250373 -1.031513  1.499214 -0.138629
2000-01-06  0.020723 -0.175899  0.457175  0.564271
2000-01-07 -0.346370  1.309142 -0.912988  1.096405
[5 rows x 4 columns]

Apply can also accept multiple axes in the axis argument. This will pass a DataFrame of the cross-section to the applied function.

In [133]: f = lambda x: (x-x.mean(1)/x.std(1))

In [134]: result = panel.apply(f, axis = ['items','major_axis'])

In [135]: result

<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC

In [136]: result.loc[:,:,'ItemA']

                   A         B         C         D
2000-01-03  0.748886 -0.323319 -1.172352  0.370451
2000-01-04 -1.594544 -1.659365 -1.444732 -0.162764
2000-01-05  0.908832 -1.220236  0.237668  0.754405
2000-01-06 -1.024669 -0.081850 -0.792957  0.641960
2000-01-07 -0.884333 -0.472889 -1.474646 -0.671871
[5 rows x 4 columns]

This is equivalent to the following

In [137]: result = Panel(dict([ (ax,f(panel.loc[:,:,ax])) for ax in panel.minor_axis ]))

In [138]: result

<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC

In [139]: result.loc[:,:,'ItemA']

                   A         B         C         D
2000-01-03  0.748886 -0.323319 -1.172352  0.370451
2000-01-04 -1.594544 -1.659365 -1.444732 -0.162764
2000-01-05  0.908832 -1.220236  0.237668  0.754405
2000-01-06 -1.024669 -0.081850 -0.792957  0.641960
2000-01-07 -0.884333 -0.472889 -1.474646 -0.671871
[5 rows x 4 columns]