BUG: pd.cut with bins=1 and input all 0s #15437

luca-s · 2017-02-17T14:35:00Z

The special case of running pd.cut() qith bins=1 an input containing all
0s raises a ValueError

closes BUG: pd.cut with bins=1 and input all 0s #15428
closes BUG: pd.qcut with q=1 and input with identical values #15431
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

jreback · 2017-02-17T14:46:17Z

pls add #15428 as a tests case as well

jreback · 2017-02-17T14:46:55Z

doc/source/whatsnew/v0.20.0.txt

@@ -587,3 +587,5 @@ Bug Fixes
 - Bug in ``Series.replace`` and ``DataFrame.replace`` which failed on empty replacement dicts (:issue:`15289`)
 - Bug in ``pd.melt()`` where passing a tuple value for ``value_vars`` caused a ``TypeError`` (:issue:`15348`)
 - Bug in ``.eval()`` which caused multiline evals to fail with local variables not on the first line (:issue:`15342`)
+


don't put whatsnew at the very end as they cause conflicts. the point of the empty space elsewhere is for you to use

Thanks, I was actually wondering what was the best place to add the new entry

jreback · 2017-02-17T14:47:07Z

doc/source/whatsnew/v0.20.0.txt

@@ -587,3 +587,5 @@ Bug Fixes
 - Bug in ``Series.replace`` and ``DataFrame.replace`` which failed on empty replacement dicts (:issue:`15289`)
 - Bug in ``pd.melt()`` where passing a tuple value for ``value_vars`` caused a ``TypeError`` (:issue:`15348`)
 - Bug in ``.eval()`` which caused multiline evals to fail with local variables not on the first line (:issue:`15342`)
+
+- Bug in ``pd.cut()`` (:issue:`15428`)


make this a useful message

jreback · 2017-02-17T14:47:30Z

pandas/tests/tools/test_tile.py

@@ -297,6 +297,10 @@ def test_single_bin(self):
        result = cut(s, 1, labels=False)
        tm.assert_series_equal(result, expected)

+        s = Series([0., 0.])


add as a new test function
add the issue number as a comment

This PR is related to that test. The test checks that pd.cut works fine when the input array contains identical values and bins=1. That is the general scenario, while this PR covers the specific case of an array contains all 0s. it would be misleading to move the test to another function.

jreback · 2017-02-17T15:16:16Z

pandas/tests/tools/test_tile.py

@@ -297,6 +297,11 @@ def test_single_bin(self):
        result = cut(s, 1, labels=False)
        tm.assert_series_equal(result, expected)

+        # issue 15428


add this for also 2 equal values that are not 0, but with a single bin, unless that is already covered

That's already covered

jreback · 2017-02-17T15:17:17Z

Regarding the issue #15428 I can add that fix here too but It's a different code bug and it's related to another function (qcut vs cut).

its the same thing. yes please add it.

luca-s · 2017-02-17T15:27:10Z

I am a little confused, it it ok now or is there something still missing?

luca-s · 2017-02-17T15:30:06Z

I am referring to #15431, do you want that issue solved in this PR too, even though it's a different bug?

jreback · 2017-02-17T15:34:23Z

@luca-s yes its virtually the same issue.

codecov-io · 2017-02-17T18:36:42Z

Codecov Report

Merging #15437 into master will decrease coverage by -0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #15437      +/-   ##
==========================================
- Coverage   91.02%   90.99%   -0.03%     
==========================================
  Files         143      143              
  Lines       49295    49295              
==========================================
- Hits        44869    44857      -12     
- Misses       4426     4438      +12

Impacted Files	Coverage Δ
pandas/tools/tile.py	`96.85% <100%> (ø)`	✅
pandas/io/gbq.py	`25% <0%> (-58.34%)`	❌
pandas/tools/merge.py	`91.76% <0%> (-0.35%)`	❌
pandas/core/common.py	`90.96% <0%> (-0.34%)`	❌
pandas/core/frame.py	`97.87% <0%> (-0.1%)`	❌

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 648ae4f...1248987. Read the comment docs.

jreback · 2017-02-18T01:13:39Z

pandas/tools/tile.py

@@ -189,6 +189,13 @@ def qcut(x, q, labels=None, retbins=False, precision=3, duplicates='raise'):
    else:
        quantiles = q
    bins = algos.quantile(x, quantiles)
+
+    # fix special case: q=1 and all identical values


this seems a bit hacky, any way to figure this out a-prior?

luca-s · 2017-02-18T13:23:00Z

What about this approach? I added an 'allow' option to _bins_to_cuts function. This might actually be useful as a qcut option too

jreback · 2017-02-18T14:10:38Z

pandas/tools/tile.py

@@ -186,9 +186,13 @@ def qcut(x, q, labels=None, retbins=False, precision=3, duplicates='raise'):

    if is_integer(q):
        quantiles = np.linspace(0, 1, q + 1)
+
+        if q == 1:


this is too narrow of a solution as well

jreback · 2017-02-20T15:42:09Z

pandas/tools/tile.py

+            raise ValueError('Cannot qcut empty array')
+
+        rng = (nanops.nanmin(x), nanops.nanmax(x))
+        if rng[0] == rng[1] and q == 1:


instead of using allow I would just adjust the quantiles right here

Sure, I can do that but this seems cleaner than increasing the edges by 0.1% for this specific case only (q=1 and identical values). That seems fine with pd.cut but not with pd.qcut, considering that we might be returning those bins (it's not hidden to the user).

Also, I thought that with a little extra work we could make 'allow' a top level pd.qcut option. Compared to 'drop' this option would maintain the total number of quantiles (some of them empty in the case of duplicated bin edges).

What do you think? If you don't like it I'll go with the 0.1% edges increment.

Any update?

can you show an example of what allow would do and why its useful? (apart from the internal use here)

The only reason it could be useful is to keep the number of quantiles as per user request. This is the only difference compare to 'drop'. I understand this might not be super useful, so it's up to you. Still I would keep the option for internal use.

@luca-s I still find this a bit odd, can you do this by simply adjusting the values (and not using this kw, even though its only internal)?

jreback · 2017-03-01T19:22:36Z

you'll want to rebase on master as travis was having some issues

The special case of running pd.cut() qith bins=1 an input containing all 0s raises a ValueError

BUG: pd.qcut with q=1 and input with identical values

luca-s · 2017-03-07T23:53:50Z

What about this new fix attempt?

jreback · 2017-03-08T00:15:54Z

@luca-s I like this a lot more!

luca-s · 2017-03-08T09:02:58Z

All good then? Or is there something missing?

jreback · 2017-03-08T13:43:26Z

thanks @luca-s

keep em coming!

The special case of running pd.cut() qith bins=1 an input containing all 0s raises a ValueError closes pandas-dev#15428 closes pandas-dev#15431 Author: Luca Scarabello <[email protected]> Author: Luca <[email protected]> Closes pandas-dev#15437 from luca-s/issue_15428 and squashes the following commits: 1248987 [Luca] rebased on master def84ba [Luca] Yet another implementation attempt 692503a [Luca Scarabello] Improved solution: using same approach as pd.cut b7d92dc [Luca] Added 'allow' duplicates option to _bins_to_cuts f56a27f [Luca Scarabello] Issue pandas-dev#15431 55806cf [Luca Scarabello] BUG: pd.cut with bins=1 and input all 0s

jreback requested changes Feb 17, 2017

View reviewed changes

jreback added Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Feb 17, 2017

luca-s force-pushed the issue_15428 branch from 6f3a721 to d842b70 Compare February 17, 2017 15:14

jreback reviewed Feb 17, 2017

View reviewed changes

jreback reviewed Feb 18, 2017

View reviewed changes

luca-s force-pushed the issue_15428 branch from 177d329 to e24c53f Compare February 18, 2017 13:29

jreback reviewed Feb 18, 2017

View reviewed changes

jreback reviewed Feb 20, 2017

View reviewed changes

luca-s force-pushed the issue_15428 branch from d79b767 to b1b230a Compare March 2, 2017 09:00

luca-s and others added 6 commits March 8, 2017 00:43

BUG: pd.cut with bins=1 and input all 0s

55806cf

The special case of running pd.cut() qith bins=1 an input containing all 0s raises a ValueError

Issue pandas-dev#15431

f56a27f

BUG: pd.qcut with q=1 and input with identical values

Added 'allow' duplicates option to _bins_to_cuts

b7d92dc

Improved solution: using same approach as pd.cut

692503a

Yet another implementation attempt

def84ba

rebased on master

1248987

luca-s force-pushed the issue_15428 branch from 8fe5222 to 1248987 Compare March 7, 2017 23:47

jreback added this to the 0.20.0 milestone Mar 8, 2017

jreback approved these changes Mar 8, 2017

View reviewed changes

jreback closed this in d32acaa Mar 8, 2017

luca-s deleted the issue_15428 branch March 8, 2017 13:49

luca-s restored the issue_15428 branch March 8, 2017 13:49

luca-s deleted the issue_15428 branch March 8, 2017 13:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: pd.cut with bins=1 and input all 0s #15437

BUG: pd.cut with bins=1 and input all 0s #15437

luca-s commented Feb 17, 2017 •

edited by jreback

Loading

jreback commented Feb 17, 2017

jreback Feb 17, 2017

luca-s Feb 17, 2017

jreback Feb 17, 2017

jreback Feb 17, 2017

luca-s Feb 17, 2017

jreback Feb 17, 2017 •

edited

Loading

luca-s Feb 17, 2017

jreback Feb 17, 2017

jreback commented Feb 17, 2017

luca-s commented Feb 17, 2017

luca-s commented Feb 17, 2017

jreback commented Feb 17, 2017

codecov-io commented Feb 17, 2017 •

edited

Loading

jreback Feb 18, 2017

luca-s commented Feb 18, 2017 •

edited

Loading

jreback Feb 18, 2017

jreback Feb 20, 2017

luca-s Feb 20, 2017

luca-s Mar 1, 2017

jreback Mar 1, 2017 •

edited

Loading

luca-s Mar 5, 2017

jreback Mar 5, 2017

jreback commented Mar 1, 2017

luca-s commented Mar 7, 2017

jreback commented Mar 8, 2017

luca-s commented Mar 8, 2017

jreback commented Mar 8, 2017

BUG: pd.cut with bins=1 and input all 0s #15437

BUG: pd.cut with bins=1 and input all 0s #15437

Conversation

luca-s commented Feb 17, 2017 • edited by jreback Loading

jreback commented Feb 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Feb 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 17, 2017

luca-s commented Feb 17, 2017

luca-s commented Feb 17, 2017

jreback commented Feb 17, 2017

codecov-io commented Feb 17, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

luca-s commented Feb 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Mar 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 1, 2017

luca-s commented Mar 7, 2017

jreback commented Mar 8, 2017

luca-s commented Mar 8, 2017

jreback commented Mar 8, 2017

luca-s commented Feb 17, 2017 •

edited by jreback

Loading

jreback Feb 17, 2017 •

edited

Loading

codecov-io commented Feb 17, 2017 •

edited

Loading

luca-s commented Feb 18, 2017 •

edited

Loading

jreback Mar 1, 2017 •

edited

Loading