Added FrozenList subtraction #15506

benthayer · 2017-02-25T20:52:16Z

closes Groupby all levels except the specified ones #15475
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

jreback · 2017-02-25T21:01:48Z

pandas/indexes/frozen.py

+    def __sub__(self, other):
+        other = set(other)
+        temp = [x for x in self if x not in other]
+        return self.__class__(temp)


add isub as well (and a test)

jreback · 2017-02-25T21:02:31Z

doc/source/whatsnew/v0.20.0.txt

@@ -26,6 +26,8 @@ Check the :ref:`API Changes <whatsnew_0200.api_breaking>` and :ref:`deprecations
 New features
 ~~~~~~~~~~~~

+- Added subtraction from FrozenLists (:issue:`15475`)


this is called difference

i think might be nice to add an example to th docs as well

codecov-io · 2017-02-25T22:47:07Z

Codecov Report

Merging #15506 into master will decrease coverage by -0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #15506      +/-   ##
==========================================
- Coverage   91.06%   91.04%   -0.03%     
==========================================
  Files         136      136              
  Lines       49099    49182      +83     
==========================================
+ Hits        44714    44779      +65     
- Misses       4385     4403      +18

Impacted Files	Coverage Δ
pandas/indexes/frozen.py	`93.24% <100%> (-0.31%)`	❌
pandas/io/gbq.py	`40% <0%> (-46.67%)`	❌
pandas/util/decorators.py	`67.27% <0%> (-1.96%)`	❌
pandas/tools/merge.py	`91.78% <0%> (-0.35%)`	❌
pandas/util/testing.py	`81.87% <0%> (-0.19%)`	❌
pandas/core/frame.py	`97.74% <0%> (-0.1%)`	❌
pandas/core/algorithms.py	`94.48% <0%> (ø)`	✅
pandas/core/categorical.py	`96.92% <0%> (+0.01%)`	✅
pandas/formats/style.py	`96.8% <0%> (+0.63%)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2340fb8...428a1b3. Read the comment docs.

jreback · 2017-02-25T22:57:10Z

doc/source/groupby.rst

@@ -126,6 +126,14 @@ We could naturally group by either the ``A`` or ``B`` columns or both:
   grouped = df.groupby('A')
   grouped = df.groupby(['A', 'B'])

+If we also have a MultiIndex on columns ``A`` and ``B``, we can group by all
+but the specified columns


add a versionadded tag here

jreback · 2017-02-25T22:57:37Z

pandas/tests/indexes/test_frozen.py

+        result = FrozenList([1, 2, 3, 2]) - [2]
+        expected = FrozenList([1, 3])
+        self.check_result(result, expected)
+
    def test_inplace(self):


can you rename this to make consistent

…ce_add for consistency

jorisvandenbossche · 2017-02-26T00:51:56Z

I want to raise some concern regarding adding this feature (without wanting to dismiss your effort @bthayer2365 !). I think we should think about this some more.

Adding this would introduce a discrepancy between index level names and column names, while we actually try to lessen the gap (eg by being able to specify an index level name in places where you can specify a column name). This is actually a feature we removed from column names, in favor of the difference method.
So that is maybe also an alternative? Putting this in a method instead of overloading subtraction? (although I certainly agree what we try to obtain here is useful, and as a method it is more verbose)

jreback · 2017-02-26T02:29:13Z

no conceptual problem with making these real setops, e.g. .union, .difference as @jorisvandenbossche suggests. Publicly these are only used for names of levels anyhow.

benthayer · 2017-02-26T02:49:04Z

Would you like me to change __add__ to union as well?

jreback · 2017-02-26T02:52:57Z

yes; u will have to deprecate the add though

benthayer · 2017-02-26T02:54:16Z

To do that, do I just decorate it with @deprecated, or is there more?

jreback · 2017-02-26T03:01:19Z

that should work (and add in whatsnew as well)
iadd too

jreback · 2017-02-27T15:53:39Z

doc/source/groupby.rst

+
+.. ipython:: python
+
+   df.set_index(['A', 'B'])


this won't work as it returns a new object, instead something like

df2 = df.set_index(['A', 'B']) grouped = df2.groupby(level=....)

jreback · 2017-02-27T15:54:02Z

doc/source/whatsnew/v0.20.0.txt

@@ -26,6 +26,8 @@ Check the :ref:`API Changes <whatsnew_0200.api_breaking>` and :ref:`deprecations
 New features
 ~~~~~~~~~~~~

+- Added difference from FrozenLists (:issue:`15475`)


.difference() method for FrozenList

jreback · 2017-02-27T15:54:44Z

pandas/indexes/frozen.py

@@ -25,6 +27,7 @@ class FrozenList(PandasObject, list):
    #           typechecks

    def __add__(self, other):
+        warnings.warn("__add__ is deprecated, use union(...)", FutureWarning)
        if isinstance(other, tuple):


return self.union(other)

jreback · 2017-02-27T15:54:53Z

pandas/indexes/frozen.py

@@ -80,6 +83,16 @@ def __repr__(self):
    __setitem__ = __setslice__ = __delitem__ = __delslice__ = _disabled
    pop = append = extend = remove = sort = insert = _disabled

+    def union(self, other):
+        if isinstance(other, tuple):


can you add a doc-string

jreback · 2017-02-27T15:54:59Z

pandas/indexes/frozen.py

+        return self.__class__(super(FrozenList, self).__add__(other))
+
+    def difference(self, other):
+        other = set(other)


jreback · 2017-02-27T15:55:53Z

pandas/tests/indexes/test_frozen.py

@@ -14,21 +14,20 @@ def setUp(self):
        self.container = FrozenList(self.lst)
        self.klass = FrozenList

-    def test_add(self):


can you add a test for assert_produces_warning(FutureWarning) for the deprecation of __add__

jreback · 2017-02-27T15:56:33Z

pandas/indexes/frozen.py

+    def union(self, other):
+        if isinstance(other, tuple):
+            other = list(other)
+        return self.__class__(super(FrozenList, self).__add__(other))


I think __iadd__ is also defined? if so, deprecate that as well (no replacement, just a deprecation)

…ion()

benthayer · 2017-03-02T05:45:28Z

Anything else? All tests passed 👍

jreback · 2017-03-02T13:25:30Z

thanks @bthayer2365

xref pandas-dev#15506

closes pandas-dev#15475 Author: Ben Thayer <[email protected]> Author: bthayer2365 <[email protected]> Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits: 428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew 84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index 8dbde1e [Ben Thayer] Rebased to upstream/master 6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 66b3b91 [Ben Thayer] Fixed issue number 3d6cee5 [Ben Thayer] Depricated __add__ in favor of union ccd75c7 [Ben Thayer] Changed __sub__ to difference cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs 79dd958 [Ben Thayer] Updated whatsnew to reflect changes 0fc7e19 [Ben Thayer] Removed whitespace 73564ab [Ben Thayer] Added FrozenList subtraction fee7a7d [bthayer2365] Merge branch 'master' into frozen-index 6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 2ab85cb [Ben Thayer] Fixed issue number cb95089 [Ben Thayer] Depricated __add__ in favor of union 2e43849 [Ben Thayer] Changed __sub__ to difference fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs cd73faa [Ben Thayer] Updated whatsnew to reflect changes f6381a8 [Ben Thayer] Removed whitespace ada7cda [Ben Thayer] Added FrozenList subtraction

xref pandas-dev#15506

cpcloud · 2017-03-02T20:36:18Z

@bthayer2365 Great job! Thanks for your contribution.

benthayer · 2017-03-03T01:57:08Z

Glad I could help!

See #15559. This temporarily reverts #15506, to see if this fixes the doc build slowdown. Author: Joris Van den Bossche <[email protected]> Closes #15566 from jorisvandenbossche/revert and squashes the following commits: befd858 [Joris Van den Bossche] Revert "ENH: Added FrozenList difference setop" 527ded9 [Joris Van den Bossche] Revert "TST: remove deprecated usages of FrozenList.__add__ from test code"

closes pandas-dev#15475 Author: Ben Thayer <[email protected]> Author: bthayer2365 <[email protected]> Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits: 428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew 84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index 8dbde1e [Ben Thayer] Rebased to upstream/master 6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 66b3b91 [Ben Thayer] Fixed issue number 3d6cee5 [Ben Thayer] Depricated __add__ in favor of union ccd75c7 [Ben Thayer] Changed __sub__ to difference cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs 79dd958 [Ben Thayer] Updated whatsnew to reflect changes 0fc7e19 [Ben Thayer] Removed whitespace 73564ab [Ben Thayer] Added FrozenList subtraction fee7a7d [bthayer2365] Merge branch 'master' into frozen-index 6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 2ab85cb [Ben Thayer] Fixed issue number cb95089 [Ben Thayer] Depricated __add__ in favor of union 2e43849 [Ben Thayer] Changed __sub__ to difference fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs cd73faa [Ben Thayer] Updated whatsnew to reflect changes f6381a8 [Ben Thayer] Removed whitespace ada7cda [Ben Thayer] Added FrozenList subtraction

xref pandas-dev#15506

closes pandas-dev#15475 Author: Ben Thayer <[email protected]> Author: bthayer2365 <[email protected]> Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits: 428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew 84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index 8dbde1e [Ben Thayer] Rebased to upstream/master 6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 66b3b91 [Ben Thayer] Fixed issue number 3d6cee5 [Ben Thayer] Depricated __add__ in favor of union ccd75c7 [Ben Thayer] Changed __sub__ to difference cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs 79dd958 [Ben Thayer] Updated whatsnew to reflect changes 0fc7e19 [Ben Thayer] Removed whitespace 73564ab [Ben Thayer] Added FrozenList subtraction fee7a7d [bthayer2365] Merge branch 'master' into frozen-index 6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union() 2ab85cb [Ben Thayer] Fixed issue number cb95089 [Ben Thayer] Depricated __add__ in favor of union 2e43849 [Ben Thayer] Changed __sub__ to difference fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency 2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs cd73faa [Ben Thayer] Updated whatsnew to reflect changes f6381a8 [Ben Thayer] Removed whitespace ada7cda [Ben Thayer] Added FrozenList subtraction

xref pandas-dev#15506

See pandas-dev#15559. This temporarily reverts pandas-dev#15506, to see if this fixes the doc build slowdown. Author: Joris Van den Bossche <[email protected]> Closes pandas-dev#15566 from jorisvandenbossche/revert and squashes the following commits: befd858 [Joris Van den Bossche] Revert "ENH: Added FrozenList difference setop" 527ded9 [Joris Van den Bossche] Revert "TST: remove deprecated usages of FrozenList.__add__ from test code"

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

Re-attempt of gh-15506. Closes gh-15475.

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

benthayer added 3 commits February 25, 2017 01:47

Added FrozenList subtraction

ada7cda

Removed whitespace

f6381a8

Updated whatsnew to reflect changes

cd73faa

jreback requested changes Feb 25, 2017

View reviewed changes

Added __isub__ and groupby example to docs

2fad2f7

jreback reviewed Feb 25, 2017

View reviewed changes

Added versionadded tag in docs and renamed test_inplace to test_inpla…

fdcfbbb

…ce_add for consistency

cpcloud self-assigned this Feb 25, 2017

cpcloud added the HackIllinois 2017 label Feb 25, 2017

cpcloud added this to the 0.20.0 milestone Feb 25, 2017

Changed __sub__ to difference

2e43849

benthayer added 2 commits February 25, 2017 21:35

Depricated __add__ in favor of union

cb95089

Fixed issue number

2ab85cb

jreback reviewed Feb 27, 2017

View reviewed changes

Added docstrings, depricated __iadd__, changed __add__ to use self.un…

6a2b48d

…ion()

benthayer force-pushed the frozen-index branch from 53789b3 to 6a2b48d Compare March 1, 2017 01:52

jreback closed this in 37fe2c4 Mar 2, 2017

jsexauer mentioned this pull request Mar 2, 2017

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

jreback added a commit to jreback/pandas that referenced this pull request Mar 2, 2017

TST: remove deprecated usages of FrozenList.__add__ from test code

cd67704

xref pandas-dev#15506

mcocdawc pushed a commit to mcocdawc/pandas that referenced this pull request Mar 2, 2017

TST: remove deprecated usages of FrozenList.__add__ from test code

86b440f

xref pandas-dev#15506

jorisvandenbossche mentioned this pull request Mar 4, 2017

Revert FrozenList changes (doc build slowdown, #15559) #15566

Closed

jreback added a commit to mcocdawc/pandas that referenced this pull request Mar 6, 2017

TST: remove deprecated usages of FrozenList.__add__ from test code

82a9e3e

xref pandas-dev#15506

AnkurDedania pushed a commit to AnkurDedania/pandas that referenced this pull request Mar 21, 2017

TST: remove deprecated usages of FrozenList.__add__ from test code

380a082

xref pandas-dev#15506

jreback mentioned this pull request Jun 7, 2017

FrozenNDArray cannot repr scalars #9031

Closed

gfyoung added a commit to forking-repos/pandas that referenced this pull request Oct 28, 2018

ENH: Add FrozenList.union and .difference

570e9e3

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

gfyoung added Enhancement API Design Frozen labels Oct 28, 2018

gfyoung mentioned this pull request Oct 28, 2018

ENH: Add FrozenList.union and .difference #23394

Merged

gfyoung added a commit to forking-repos/pandas that referenced this pull request Nov 1, 2018

ENH: Add FrozenList.union and .difference

0468d9a

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

gfyoung added a commit to forking-repos/pandas that referenced this pull request Nov 1, 2018

ENH: Add FrozenList.union and .difference

c1a375e

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

jreback pushed a commit that referenced this pull request Nov 3, 2018

ENH: Add FrozenList.union and .difference (#23394)

06c8643

Re-attempt of gh-15506. Closes gh-15475.

gfyoung mentioned this pull request Nov 4, 2018

DOC: Fix syntax error in groupby docs #23498

Merged

JustinZhengBC pushed a commit to JustinZhengBC/pandas that referenced this pull request Nov 14, 2018

ENH: Add FrozenList.union and .difference (pandas-dev#23394)

126edd9

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

ENH: Add FrozenList.union and .difference (pandas-dev#23394)

631e48d

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: Add FrozenList.union and .difference (pandas-dev#23394)

7dc4edb

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: Add FrozenList.union and .difference (pandas-dev#23394)

1e71775

Re-attempt of pandas-devgh-15506. Closes pandas-devgh-15475.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added FrozenList subtraction #15506

Added FrozenList subtraction #15506

benthayer commented Feb 25, 2017 •

edited

Loading

jreback Feb 25, 2017

jreback Feb 25, 2017

codecov-io commented Feb 25, 2017 •

edited

Loading

jreback Feb 25, 2017

jreback Feb 25, 2017

jorisvandenbossche commented Feb 26, 2017

jreback commented Feb 26, 2017

benthayer commented Feb 26, 2017

jreback commented Feb 26, 2017

benthayer commented Feb 26, 2017

jreback commented Feb 26, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017

jreback Feb 27, 2017 •

edited

Loading

benthayer commented Mar 2, 2017

jreback commented Mar 2, 2017

cpcloud commented Mar 2, 2017

benthayer commented Mar 3, 2017

Added FrozenList subtraction #15506

Added FrozenList subtraction #15506

Conversation

benthayer commented Feb 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Feb 25, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Feb 26, 2017

jreback commented Feb 26, 2017

benthayer commented Feb 26, 2017

jreback commented Feb 26, 2017

benthayer commented Feb 26, 2017

jreback commented Feb 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Feb 27, 2017 • edited Loading

Choose a reason for hiding this comment

benthayer commented Mar 2, 2017

jreback commented Mar 2, 2017

cpcloud commented Mar 2, 2017

benthayer commented Mar 3, 2017

benthayer commented Feb 25, 2017 •

edited

Loading

codecov-io commented Feb 25, 2017 •

edited

Loading

jreback Feb 27, 2017 •

edited

Loading