You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When attempting to read in a .XPT file downloaded from the CDC website using pandas.read_sas, a large number of PerformanceWarnings is printed. The file that causes the warnings to appear is this one: https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DR1TOT_J.XPT .
```:1: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()````
I'm not familiar with the inner workings of read_sas but I can only assume that it's creating the DataFrame in a way that is causing another part of pandas to complain.
Thanks a lot!
Expected Behavior
The expected behavior is for the file to be read in without a large number of warnings being printed. :)
Installed Versions
INSTALLED VERSIONS
commit : ca60aab
python : 3.9.13.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.16.3-microsoft-standard-WSL2
Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
Hi there,
When attempting to read in a
.XPT
file downloaded from the CDC website usingpandas.read_sas
, a large number of PerformanceWarnings is printed. The file that causes the warnings to appear is this one: https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DR1TOT_J.XPT .```:1: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling
frame.insert
many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()````The above warning is repeated 68 times. A different file that doesn't produce any warnings is: https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DEMO_J.XPT
I'm not familiar with the inner workings of
read_sas
but I can only assume that it's creating the DataFrame in a way that is causing another part of pandas to complain.Thanks a lot!
Expected Behavior
The expected behavior is for the file to be read in without a large number of warnings being printed. :)
Installed Versions
INSTALLED VERSIONS
commit : ca60aab
python : 3.9.13.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.16.3-microsoft-standard-WSL2
Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.4.4
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 65.3.0
pip : 22.2.2
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : 2022.8.2
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.3
numba : None
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : 1.4.41
tables : 3.6.1
tabulate : 0.8.10
xarray : 2022.6.0
xlrd : None
xlwt : None
zstandard : None
The text was updated successfully, but these errors were encountered: