Skip to content

BUG: Large number of PerformanceWarnings when attempting to read .xpt file #48595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
leeping opened this issue Sep 16, 2022 · 1 comment · Fixed by #48603
Closed
2 of 3 tasks

BUG: Large number of PerformanceWarnings when attempting to read .xpt file #48595

leeping opened this issue Sep 16, 2022 · 1 comment · Fixed by #48603
Labels
Bug IO SAS SAS: read_sas Performance Memory or execution speed performance

Comments

@leeping
Copy link

leeping commented Sep 16, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
diet = pd.read_sas('DR1TOT_J.XPT',index='SEQN')

Issue Description

Hi there,

When attempting to read in a .XPT file downloaded from the CDC website using pandas.read_sas, a large number of PerformanceWarnings is printed. The file that causes the warnings to appear is this one: https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DR1TOT_J.XPT .

```:1: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()````

The above warning is repeated 68 times. A different file that doesn't produce any warnings is: https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DEMO_J.XPT

I'm not familiar with the inner workings of read_sas but I can only assume that it's creating the DataFrame in a way that is causing another part of pandas to complain.

Thanks a lot!

Expected Behavior

The expected behavior is for the file to be read in without a large number of warnings being printed. :)

Installed Versions

INSTALLED VERSIONS

commit : ca60aab
python : 3.9.13.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.16.3-microsoft-standard-WSL2
Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.4
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 65.3.0
pip : 22.2.2
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : 2022.8.2
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.3
numba : None
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : 1.4.41
tables : 3.6.1
tabulate : 0.8.10
xarray : 2022.6.0
xlrd : None
xlwt : None
zstandard : None

@leeping leeping added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 16, 2022
@phofl phofl added Performance Memory or execution speed performance IO SAS SAS: read_sas and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 17, 2022
@leeping
Copy link
Author

leeping commented Sep 17, 2022

Thank you very much for looking at this. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug IO SAS SAS: read_sas Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants