ENH: Allow where/mask/Indexers to accept callable #12539

sinhrks · 2016-03-06T02:02:31Z

closes Allow .where to accept callable as condition #12533
closes API: chained reshaping ops #11485
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

Current impl is very simple like assign. Another idea is applying some restriction to return value from callable (for example, only allow aligned DataFrame or scalar).

kawochen · 2016-03-06T04:15:54Z

One potential problem is when you subclass DataFrame and make it callable.

sinhrks · 2016-03-06T05:12:22Z

Do you know such a package? The logic is not specific here and I don't think we have to care ATM.

kawochen · 2016-03-06T14:46:21Z

I don't know anyone that uses it that way. But since the order of consideration is relevant, I think it should be documented that the args are first tested for callability.

jreback · 2016-03-06T14:54:52Z

let's simultaneously do all of the chained reshaping ops #11485 in 0.18.1, making them consistent.

sinhrks · 2016-03-09T10:47:11Z

Sure, once done and changed the title.

jreback · 2016-03-09T18:42:36Z

doc/source/whatsnew/v0.18.1.txt

+
+.. ipython:: python
+
+   # callable returns bool indexer


need to have some tests if people try to pass something like:

df.loc[lambda x: ...., list_of_columns] which is a bit odd, but I suppose ok, otherwise we need to interpret this

OK, added some tests for mixture of lamda and scalar/list.

jreback · 2016-03-15T13:50:23Z

doc/source/whatsnew/v0.18.1.txt

+Method chaininng improvements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Following methods / ``Indexer`` now accept ``callable`` as input. It is


The following methods / indexers now acccept a callable

codecov-io · 2016-03-16T17:31:35Z

Current coverage is 83.91%

Merging #12539 into master will increase coverage by +<.01%

@@             master     #12539   diff @@
==========================================
  Files           136        136          
  Lines         49908      49933    +25   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41901    +25   
  Misses         8032       8032          
  Partials          0          0

Powered by Codecov. Last updated by 3921b32

TomAugspurger · 2016-04-19T01:02:13Z

doc/source/whatsnew/v0.18.1.txt

+- ``[]`` indexing
+
+When a ``callable`` is passed, it is evaluated on the caller
+``Series/DataFrame`` prior to any other logics.


What do you mean by this?

jreback · 2016-04-25T14:41:15Z

ok, how about for .query() we than raise if the input is NOT a string with a nice mesage (to use .loc)? or could show a warning and just do it anyhow.

jorisvandenbossche · 2016-04-26T22:38:20Z

You already get an error when passing a callable to query, but I agree a nicer message can be raised.

shoyer · 2016-04-26T22:49:28Z

I would definitely raise if the input to .query() is not a string. Adding warnings and then doing something anyways is a bad idea.

jreback · 2016-04-26T22:56:23Z

ok @sinhrks if you can make those mods would be great. (you might need to adjust docs/example as I think you use .query, but can just change to .loc)

sinhrks · 2016-04-28T12:54:13Z

Thanks for all the feedbacks. Updated. Lmk if I missed something.

jreback · 2016-04-28T13:21:42Z

pandas/tests/indexing/test_indexing.py

@@ -5489,6 +5489,269 @@ def test_none_coercion_mixed_dtypes(self):
                                        strict_nan=True)


+class TestIndexingCallable(tm.TestCase):


I would make this a separate file: indexing/test_callables.py

jreback · 2016-04-28T13:22:17Z

just minor change otherwise lgtm.

@jorisvandenbossche @shoyer

jorisvandenbossche · 2016-04-28T13:31:40Z

doc/source/whatsnew/v0.18.1.txt

+   df = pd.DataFrame({'A': [1, 2, 3],
+                      'B': [4, 5, 6],
+                      'C': [7, 8, 9]})
+   df.where(lambda x: x > 4, lambda x: x + 10)


I would add this example somewhere in the where docs as well (http://pandas-docs.github.io/pandas-docs-travis/indexing.html#the-where-method-and-masking)

jorisvandenbossche · 2016-04-28T13:32:18Z

@sinhrks Can you also update the docstrings for loc/iloc/ix (eg for loc https://github.com/pydata/pandas/blob/65ed3afd092908130905e7d86ff1b6b9c2002d00/pandas/core/indexing.py#L1314), these are used in the API docs.

For the rest, +1!

jreback · 2016-04-29T13:59:40Z

pandas/core/frame.py

@@ -2454,8 +2459,9 @@ def assign(self, **kwargs):
        kwargs : keyword, value pairs
            keywords are the column names. If the values are
            callable, they are computed on the DataFrame and
-            assigned to the new columns. If the values are
-            not callable, (e.g. a Series, scalar, or array),
+            assigned to the new columns. The callable must not


add a versionadded tag here (and other doc-strings)

assign impl is unchanged.

jreback · 2016-04-29T14:01:04Z

@sinhrks some very minor doc-changes. ping when pushed.

jreback · 2016-04-29T17:06:14Z

thanks @sinhrks !

awesome enhancement!

sinhrks added Indexing Related to indexing on series/frames, not to indexes themselves API Design labels Mar 6, 2016

sinhrks added this to the 0.18.1 milestone Mar 6, 2016

jreback mentioned this pull request Mar 6, 2016

API: chained reshaping ops #11485

Closed

2 tasks

sinhrks force-pushed the where branch 4 times, most recently from 9b39b6f to f2430c4 Compare March 9, 2016 08:20

sinhrks changed the title ~~ENH: Allow .where to accept callable as condition~~ ENH: Allow where/mask/query/Indexers to accept callable Mar 9, 2016

jreback reviewed Mar 9, 2016
View reviewed changes

sinhrks force-pushed the where branch from f2430c4 to 3f98de3 Compare March 14, 2016 06:59

jreback reviewed Mar 15, 2016
View reviewed changes

sinhrks force-pushed the where branch from 3f98de3 to fd6c69d Compare March 16, 2016 14:53

TomAugspurger reviewed Apr 19, 2016
View reviewed changes

sinhrks force-pushed the where branch 5 times, most recently from 7ad04b7 to ebe5aaf Compare April 27, 2016 21:29

sinhrks changed the title ~~ENH: Allow where/mask/query/Indexers to accept callable~~ ENH: Allow where/mask/Indexers to accept callable Apr 27, 2016

sinhrks force-pushed the where branch from ebe5aaf to 8795ba1 Compare April 28, 2016 11:31

jreback reviewed Apr 28, 2016
View reviewed changes

jorisvandenbossche reviewed Apr 28, 2016
View reviewed changes

sinhrks force-pushed the where branch from 8795ba1 to b52d5d9 Compare April 28, 2016 21:33

jreback reviewed Apr 29, 2016
View reviewed changes

ENH: Allow .where to accept callable as condition

6b5d618

sinhrks force-pushed the where branch from b52d5d9 to 6b5d618 Compare April 29, 2016 16:56

jreback closed this in 7bbd031 Apr 29, 2016

sinhrks deleted the where branch April 29, 2016 17:06

jreback mentioned this pull request May 24, 2016

API/ENH: enable boolean indexing conditions to reference its input frame via eval #5501

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Allow where/mask/Indexers to accept callable #12539

ENH: Allow where/mask/Indexers to accept callable #12539

sinhrks commented Mar 6, 2016 •

edited

Loading

kawochen commented Mar 6, 2016

sinhrks commented Mar 6, 2016

kawochen commented Mar 6, 2016

jreback commented Mar 6, 2016

sinhrks commented Mar 9, 2016

jreback Mar 9, 2016

sinhrks Mar 15, 2016

jreback Mar 15, 2016

codecov-io commented Mar 16, 2016 •

edited

Loading

TomAugspurger Apr 19, 2016

jreback commented Apr 25, 2016

jorisvandenbossche commented Apr 26, 2016

shoyer commented Apr 26, 2016

jreback commented Apr 26, 2016

sinhrks commented Apr 28, 2016

jreback Apr 28, 2016

jreback commented Apr 28, 2016

jorisvandenbossche Apr 28, 2016

jorisvandenbossche commented Apr 28, 2016

jreback Apr 29, 2016

sinhrks Apr 29, 2016 •

edited

Loading

jreback commented Apr 29, 2016

jreback commented Apr 29, 2016

		@@ -5489,6 +5489,269 @@ def test_none_coercion_mixed_dtypes(self):
		strict_nan=True)


		class TestIndexingCallable(tm.TestCase):

ENH: Allow where/mask/Indexers to accept callable #12539

ENH: Allow where/mask/Indexers to accept callable #12539

Conversation

sinhrks commented Mar 6, 2016 • edited Loading

kawochen commented Mar 6, 2016

sinhrks commented Mar 6, 2016

kawochen commented Mar 6, 2016

jreback commented Mar 6, 2016

sinhrks commented Mar 9, 2016

jreback Mar 9, 2016

Choose a reason for hiding this comment

sinhrks Mar 15, 2016

Choose a reason for hiding this comment

jreback Mar 15, 2016

Choose a reason for hiding this comment

codecov-io commented Mar 16, 2016 • edited Loading

Current coverage is 83.91%

TomAugspurger Apr 19, 2016

Choose a reason for hiding this comment

jreback commented Apr 25, 2016

jorisvandenbossche commented Apr 26, 2016

shoyer commented Apr 26, 2016

jreback commented Apr 26, 2016

sinhrks commented Apr 28, 2016

jreback Apr 28, 2016

Choose a reason for hiding this comment

jreback commented Apr 28, 2016

jorisvandenbossche Apr 28, 2016

Choose a reason for hiding this comment

jorisvandenbossche commented Apr 28, 2016

jreback Apr 29, 2016

Choose a reason for hiding this comment

sinhrks Apr 29, 2016 • edited Loading

Choose a reason for hiding this comment

jreback commented Apr 29, 2016

jreback commented Apr 29, 2016

sinhrks commented Mar 6, 2016 •

edited

Loading

codecov-io commented Mar 16, 2016 •

edited

Loading

sinhrks Apr 29, 2016 •

edited

Loading