Skip to content

Detect CPORT header in SAS files #44300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 6, 2021
Merged

Detect CPORT header in SAS files #44300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5b43e1b
Detect CPORT header in SAS files
lordgrenville Nov 3, 2021
8485358
add unit test for SAS CPORT check
lordgrenville Nov 4, 2021
05f379d
add reference to documentation
lordgrenville Nov 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pandas/io/sas/sas_xport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,12 @@ def _read_header(self):
# read file header
line1 = self._get_row()
if line1 != _correct_line1:
if "**COMPRESSED**" in line1:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this value in any file format document provided by SAS? If so, would be good to include a reference.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I've added it in a new commit

# this was created with the PROC CPORT method and can't be read
# https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/movefile/p1bm6aqp3fw4uin1hucwh718f6kp.htm
raise ValueError(
"Header record indicates a CPORT file, which is not readable."
)
raise ValueError("Header record is not an XPORT file.")

line2 = self._get_row()
Expand Down
Binary file added pandas/tests/io/sas/data/DEMO_PUF.cpt
View file
Open in desktop
Binary file not shown.
9 changes: 9 additions & 0 deletions pandas/tests/io/sas/test_xport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ def setup_method(self, datapath):
self.file02 = os.path.join(self.dirpath, "SSHSV1_A.xpt")
self.file03 = os.path.join(self.dirpath, "DRXFCD_G.xpt")
self.file04 = os.path.join(self.dirpath, "paxraw_d_short.xpt")
self.file05 = os.path.join(self.dirpath, "DEMO_PUF.cpt")

with td.file_leak_context():
yield
Expand Down Expand Up @@ -157,3 +158,11 @@ def test_truncated_float_support(self):

data = read_sas(self.file04, format="xport")
tm.assert_frame_equal(data.astype("int64"), data_csv)

def test_cport_header_found_raises(self):
# Test with DEMO_PUF.cpt, the beginning of puf2019_1_fall.xpt
# from https://www.cms.gov/files/zip/puf2019.zip
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this public domain or similar? If not, might be good to have someone create a CPT file from scratch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's publicly available data (from the US government's CMS, stored at https://www.cms.gov/files/zip/puf2019.zip). I don't have SAS so can't create a file. But the compressed CPORT files will have **COMPRESSED** **COMPRESSED** **COMPRESSED** **COM as the first 40 characters (as documented here), so I just opened this file in a text editor and deleted everything after the first few lines.

# (despite the extension, it's a cpt file)
msg = "Header record indicates a CPORT file, which is not readable."
with pytest.raises(ValueError, match=msg):
read_sas(self.file05, format="xport")