BUG: Problem importing compressed dataset (with repro datasets)

[rxxg]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
# Read 900-line compressed dataset correctly
pd.read_sas('date_working.sas7bdat')
# Read 1000-line uncompressed dataset correctly
pd.read_sas('date_uncomp.sas7bdat')
# Read 1000-line compressed dataset - crashes (and leaves file open ...)
pd.read_sas('date_broken.sas7bdat')

Problem description

I have a 1000-row SAS dataset (written by SAS 9.4 TS Level 1M6 on X64_SRV16 platform) which pandas chokes on reading. I attach the problematic dataset, as well as the same dataset uncompressed and a smaller subset of the compressed dataset that can be correctly read.

datasets.zip

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sasreader.py", line 131, in read_sas
data = reader.read()
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sas7bdat.py", line 695, in read
rslt = self._chunk_to_dataframe()
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sas7bdat.py", line 739, in _chunk_to_dataframe
rslt[name] = self._byte_chunk[jb, :].view(dtype=self.byte_order + "d")
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3037, in setitem
self._set_item(key, value)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3113, in _set_item
value = self._sanitize_column(key, value)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3758, in _sanitize_column
value = sanitize_index(value, self.index)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\internals\construction.py", line 748, in sanitize_index
"Length of values "
ValueError: Length of values (1000) does not match length of index (527)

Expected Output

No crash

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.7.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.17763
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 1.1.0
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 10.0.1
setuptools : 39.0.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
numba : None

[rxxg]

Upstream project can correctly read all datasets.

[rxxg]

I take it back, both projects drop lines between 1998-03-28 and 1999-07-15 (rows 452 and 926).

[TomAugspurger]

@rxxg sounds like the fix would need to go in sas7bdat then? Have you submitted an issue there? Will you post it here when you do?

[rxxg]

@TomAugspurger I don't know the relationship between pandas and sas7bdat (the pandas header just says that sas7bdat.py is 'based on code written by Jared Hobbs'), but I can certainly file a bug with that project as well. It doesn't look like there has been substantial work on it for a while, though.

[TomAugspurger]

Ah, my apologies. I thought we used sas7bdat, but apparently not. Are you interested in investigating more?

[rxxg]

Umm ... I already had a very quick look, but I don't think I will be able to devote the time it would take.

[paul-lilley]

Hi, I may be able to have a look at this at the same time as https://github.com/pandas-dev/pandas/issues/31243. I'm waiting for a response there first.

[ofajardo]

just in case pyreadstat can read all of these files without issues. You could consider adding it as a backend for read_sas (it is already used for read_spss)

[jonashaag]

@rxxg I found a fix to your issue. Is it fine to add your test file to the Pandas source code?

[jonashaag]

@rxxg I'm going to go ahead and add your test case to #47115. Please let me know if you don't want it in the Pandas source code.

[rxxg]

It's fine to include the test data, thanks for your time!