Skip to content

Boolean indexing and assignment from python list broken #19406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Dobatymo opened this issue Jan 26, 2018 · 6 comments · Fixed by #37840
Closed

Boolean indexing and assignment from python list broken #19406

Dobatymo opened this issue Jan 26, 2018 · 6 comments · Fixed by #37840
Assignees
Labels
good first issue Indexing Related to indexing on series/frames, not to indexes themselves Needs Tests Unit test(s) needed to prevent regressions

Comments

@Dobatymo
Copy link
Contributor

Dobatymo commented Jan 26, 2018

There is a problem with assigning lists to indexed Series

from math import nan
import pandas as pd
s1 = pd.Series([nan, nan, "c"])
s1[s1.isnull()] = ["a","b"]
s2 = pd.Series([nan, "b", nan])
s2[s2.isnull()] = ["a","c"]
#s1: "a", "b", "c"
#s2: "a", "b", "a"

s1 shows the correct result, whereas s2 is obviously wrong. In pandas 0.19.2 the result is correct, but in 0.21.0 and 0.23.0.dev0+97.g24c07b075 I get the above behavior. It can be fixed if a numpy array instead of a python list is assigned.

from math import nan
import pandas as pd
import numpy as np
s1 = pd.Series([nan, nan, "c"])
s1[s1.isnull()] = np.array(["a","b"])
s2 = pd.Series([nan, "b", nan])
s2[s2.isnull()] = np.array(["a","c"])
#s1: "a", "b", "c"
#s2: "a", "b", "c"

There probably should be a test case to catch that...

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_US
LOCALE: None.None

pandas: 0.23.0.dev0+97.g24c07b075
pytest: 3.2.3
pip: 9.0.1
setuptools: 38.2.4
Cython: 0.27.3
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.1.13
pymysql: 0.7.11.None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@mroeschke mroeschke added Bug Indexing Related to indexing on series/frames, not to indexes themselves labels Nov 23, 2018
@phofl
Copy link
Member

phofl commented Nov 11, 2020

This works now as intended. May need some tests

@phofl phofl added good first issue Needs Tests Unit test(s) needed to prevent regressions and removed Bug labels Nov 11, 2020
@Dobatymo
Copy link
Contributor Author

Can confirm it's working with pandas 1.1.3

@paxcodes
Copy link
Contributor

take

@paxcodes
Copy link
Contributor

I'm trying to decide where to put this test. I'm looking at pandas/tests/indexing folder and thinking of putting it pandas/tests/indexing/test_na_indexing.py...?

@phofl
Copy link
Member

phofl commented Nov 14, 2020

That would be better suited in the boolean group. Isnull is here just a tool to create a boolean mask. Could you maybe search for related tests? Maybe in series/indexing/test_setitem?

@paxcodes
Copy link
Contributor

Isnull is here just a tool to create a boolean mask.

I just realized this while I was typing out the test. 😅

Will do and thanks for the guidance!

paxcodes pushed a commit to paxcodes/pandas that referenced this issue Nov 14, 2020
MarcoGorelli pushed a commit that referenced this issue Nov 15, 2020
* TST: Add test for setting item using python list (#19406)

* CLN: Change test and var name, hardcode values (#37840)

* TST: Use different types of mask (#37840)

* CLN: Parametrize mask constructor instead of mask values (#37840)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Indexing Related to indexing on series/frames, not to indexes themselves Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants