Skip to content

BUG: Construction of DataFrame works with empty list but not empty array #46822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
pavelkomarov opened this issue Apr 21, 2022 · 3 comments · Fixed by #47192
Closed
2 of 3 tasks

BUG: Construction of DataFrame works with empty list but not empty array #46822

pavelkomarov opened this issue Apr 21, 2022 · 3 comments · Fixed by #47192
Labels
API - Consistency Internal Consistency of API/Behavior Bug Constructors Series/DataFrame/Index/pd.array Constructors DataFrame DataFrame data structure Error Reporting Incorrect or improved errors from pandas Testing pandas testing functions or related to the test suite
Milestone

Comments

@pavelkomarov
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pandas.DataFrame(data=[], columns=['a','b']) # works fine
df = pandas.DataFrame(data=numpy.array([]), columns=['a','b'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 694, in __init__
    mgr = ndarray_to_mgr(
  File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 351, in ndarray_to_mgr
    _check_values_indices_shape_match(values, index, columns)
  File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 418, in _check_values_indices_shape_match
    raise ValueError("Empty data passed with indices specified.")
ValueError: Empty data passed with indices specified.

Issue Description

Constructing an empty dataframe with an empty list works fine, but giving an empty numpy array, which intuitively should behave the same as with an empty list, results in a mysterious ValueError: Empty data passed with indices specified.

Expected Behavior

Should create an empty dataframe with whatever columns listed.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.9.7.final.0
python-bits : 64
OS : Darwin
OS-release : 21.4.0
Version : Darwin Kernel Version 21.4.0: Mon Feb 21 20:34:37 PST 2022; root:xnu-8020.101.4~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.2
numpy : 1.21.4
pytz : 2021.1
dateutil : 2.8.2
pip : 21.3.1
setuptools : 57.4.0
Cython : 0.29.24
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.1
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.25.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.0.1
matplotlib : 3.4.3
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.7
pandas_gbq : 0.15.0
pyarrow : 5.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.1
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
zstandard : None

@pavelkomarov pavelkomarov added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 21, 2022
@phofl
Copy link
Member

phofl commented Apr 22, 2022

Hi, thanks for your report.

I think this works as intended. A list does not have a shape while a numpy array does, hence we get a missmatch. The error message is definitely intended

@phofl phofl added the DataFrame DataFrame data structure label Apr 22, 2022
@pavelkomarov
Copy link
Author

pavelkomarov commented Apr 22, 2022

mmmmmm. If the shape is (0,), then it makes sense just to treat it as an empty. Even if an empty list doesn't explicitly have this shape, an array([]) does, so they are intuitively equivalent. If you give a list of the wrong length or an array of the wrong shape, or a list or array or the right length, the treatment is analogous, so why wouldn't it be in the empty case? Just because one has an explicit .shape and the other doesn't shouldn't really matter.

@simonjayhawkins simonjayhawkins added Constructors Series/DataFrame/Index/pd.array Constructors API - Consistency Internal Consistency of API/Behavior and removed Needs Triage Issue that has not been reviewed by a pandas team member labels May 28, 2022
@jreback
Copy link
Contributor

jreback commented Jun 5, 2022

yeah i agree this works as intended. if you are passing an array then it must match in shape to the passed columns. yes i understand this is special casing a tiny bit, but it helps prevent errors.

@jreback jreback added this to the 1.5 milestone Jun 11, 2022
@jreback jreback added Testing pandas testing functions or related to the test suite Error Reporting Incorrect or improved errors from pandas labels Jun 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API - Consistency Internal Consistency of API/Behavior Bug Constructors Series/DataFrame/Index/pd.array Constructors DataFrame DataFrame data structure Error Reporting Incorrect or improved errors from pandas Testing pandas testing functions or related to the test suite
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants