SAS chunksize / iteration issues #14743
Conversation
| @@ -65,6 +65,32 @@ def test_from_iterator(self): | |||
| df = rdr.read(3) | |||
| tm.assert_frame_equal(df, df0.iloc[2:5, :]) | |||
|
|
|||
| def test_iterator_loop(self): | |||
| for j in 0, 1: | |||
There was a problem hiding this comment.
can you add the issue references here
| y = 0 | ||
| for x in rdr: | ||
| y += x.shape[0] | ||
| assert(y == rdr.row_count) |
|
pls add to the whatsnew. ping when ready / green. |
|
Question: this is not a problem in the xport reader? If so, is there a test that confirms this? |
|
It works there but I added a few tests. |
|
@kshedden Thanks. You can move the whatsnew notice to the 0.19.2 file |
| fname = os.path.join(self.dirpath, "test%d.sas7bdat" % k) | ||
| with open(fname, 'rb') as f: | ||
| byts = f.read() | ||
| buf = io.BytesIO(byts) |
There was a problem hiding this comment.
Is there a reason you read the file into a bytes object and not just pass the fname to read_sas ?
There was a problem hiding this comment.
It seems to work either way so I took all but one of the bytesio tests and simplified to a direct file read.
kshedden
force-pushed
the
sas_iterator
branch
from
18c1c75 to
8c1e17e
Compare
Current coverage is 85.22% (diff: 0.00%)@@ master #14743 diff @@
==========================================
Files 143 143
Lines 50807 50857 +50
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 43297 43344 +47
- Misses 7510 7513 +3
Partials 0 0
|
| with open(fname, 'rb') as f: | ||
| byts = f.read() | ||
| buf = io.BytesIO(byts) | ||
| df = pd.read_sas(buf, format="sas7bdat", encoding='utf-8') |
There was a problem hiding this comment.
I think it is more logical to keep the buffer reading here, as this test is called "test_from_buffer" is I suppose is exactly testing this (and then maybe use plain reading from file in "test_iterator_read_too_much")
|
@kshedden Thanks for the quick fix! |
closes #14734
closes #13654
git diff upstream/master | flake8 --diff