BUG: Fix the error when reading the compressed UTF-16 file #18091
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Licht-T
force-pushed
the
fix-read-zipped-utf-16-file
branch
from
a327715 to
2f29a61
Compare
Licht-T
force-pushed
the
fix-read-zipped-utf-16-file
branch
from
2f29a61 to
ae3d0d0
Compare
jreback
requested changes
Nov 3, 2017
| @@ -684,6 +684,12 @@ cdef class TextReader: | |||
| else: | |||
| raise ValueError('Unrecognized compression type: %s' % | |||
| self.compression) | |||
|
|
|||
| if b'utf-16' in (self.encoding or b''): | |||
There was a problem hiding this comment.
can you add a comment here on what is going on
pandas/io/parsers.py
Outdated
| @@ -1671,7 +1671,8 @@ def __init__(self, src, **kwds): | |||
|
|
|||
| ParserBase.__init__(self, kwds) | |||
|
|
|||
| if 'utf-16' in (kwds.get('encoding') or ''): | |||
| if kwds.get('compression') is None \ | |||
There was a problem hiding this comment.
Also, use parentheses instead of the backslash to wrap multi-line conditional.
pandas/tests/io/parser/common.py
Outdated
| @@ -750,6 +750,15 @@ def test_utf16_example(self): | |||
| result = self.read_table(buf, encoding='utf-16') | |||
| assert len(result) == 50 | |||
|
|
|||
| def test_compressed_utf16_example(self): | |||
| path = tm.get_data_path('utf16_ex.zip') | |||
There was a problem hiding this comment.
can you add the issue number here
Codecov Report
@@ Coverage Diff @@
## master #18091 +/- ##
==========================================
- Coverage 91.27% 91.26% -0.02%
==========================================
Files 163 163
Lines 50120 50120
==========================================
- Hits 45749 45740 -9
- Misses 4371 4380 +9
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #18091 +/- ##
==========================================
- Coverage 91.25% 91.23% -0.02%
==========================================
Files 163 163
Lines 50120 50120
==========================================
- Hits 45737 45728 -9
- Misses 4383 4392 +9
Continue to review full report at Codecov.
|
Licht-T
force-pushed
the
fix-read-zipped-utf-16-file
branch
from
ae3d0d0 to
ea8d6bb
Compare
|
Thanks @jreback, fixed! |
jreback
requested changes
Nov 3, 2017
doc/source/whatsnew/v0.21.1.txt
Outdated
| @@ -76,7 +76,7 @@ I/O | |||
| ^^^ | |||
|
|
|||
| - Bug in class:`~pandas.io.stata.StataReader` not converting date/time columns with display formatting addressed (:issue:`17990`). Previously columns with display formatting were normally left as ordinal numbers and not converted to datetime objects. | |||
|
|
|||
| - Bug in :func:`read_table` when reading the compressed UTF-16 file (:issue:`18071`) | |||
There was a problem hiding this comment.
can you change to read_csv which is the common spelling here
pandas/_libs/parsers.pyx
Outdated
|
|
||
| if b'utf-16' in (self.encoding or b''): | ||
| # if source is utf-16, convert source to utf-8 | ||
| source = com.UTF8Recoder(source, self.encoding.decode('utf-8')) |
There was a problem hiding this comment.
can you add a short 'why' we are doing this?
|
@Licht-T : Don't forget to add a test and check that the Python parser doesn't need patching either. |
|
@gfyoung I added the test in [pandas] python3 -m pytest pandas -k test_compressed_utf16_example -v 2:50:49 ☁ fix-read-zipped-utf-16-file ☀
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.6.2, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /usr/local/opt/python3/bin/python3.6
cachedir: .cache
metadata: {'Python': '3.6.2', 'Platform': 'Darwin-15.4.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.2.3', 'py': '1.4.34', 'pluggy': '0.4.0'}, 'Plugins': {'metadata': '1.5.0', 'html': '1.15.2'}}
rootdir: /Users/rito/GitHub/pandas, inifile: setup.cfg
plugins: metadata-1.5.0, html-1.15.2
collected 15953 items / 2 skipped
pandas/tests/io/parser/test_parsers.py::TestCParserHighMemory::test_compressed_utf16_example <- pandas/tests/io/parser/common.py PASSED
pandas/tests/io/parser/test_parsers.py::TestCParserLowMemory::test_compressed_utf16_example <- pandas/tests/io/parser/common.py PASSED
pandas/tests/io/parser/test_parsers.py::TestPythonParser::test_compressed_utf16_example <- pandas/tests/io/parser/common.py PASSED
======================================================================== 15950 tests deselected =========================================================================
======================================================== 3 passed, 2 skipped, 15950 deselected in 16.35 seconds =========================================================
|
Licht-T
force-pushed
the
fix-read-zipped-utf-16-file
branch
from
ea8d6bb to
9da8edd
Compare
|
@jreback Fixed! |
gfyoung
reviewed
Nov 3, 2017
pandas/tests/io/parser/common.py
Outdated
| expected = self.read_table(expected_path, encoding='utf-16') | ||
|
|
||
| tm.assert_frame_equal(result, expected) | ||
|
|
There was a problem hiding this comment.
- Move this test to
compression.py( same directory) - Explicitly construct the expected table instead of reading it via a text file
gfyoung
reviewed
Nov 3, 2017
doc/source/whatsnew/v0.21.1.txt
Outdated
| @@ -76,7 +76,7 @@ I/O | |||
| ^^^ | |||
|
|
|||
| - Bug in class:`~pandas.io.stata.StataReader` not converting date/time columns with display formatting addressed (:issue:`17990`). Previously columns with display formatting were normally left as ordinal numbers and not converted to datetime objects. | |||
|
|
|||
| - Bug in :func:`read_csv` when reading the compressed UTF-16 file (:issue:`18071`) | |||
There was a problem hiding this comment.
"the compressed UTF-16" --> "a compressed UTF-16 encoded"
Licht-T
force-pushed
the
fix-read-zipped-utf-16-file
branch
from
9da8edd to
b2a3f97
Compare
|
Thanks @gfyoung, fixed! |
gfyoung
reviewed
Nov 3, 2017
pandas/_libs/parsers.pyx
Outdated
| @@ -374,6 +374,17 @@ cdef class TextReader: | |||
| float_precision=None, | |||
| skip_blank_lines=True): | |||
|
|
|||
| # encoding | |||
There was a problem hiding this comment.
I know that you copied and pasted this, but let's take this opportunity to provide a much-more informative comment about this whole block of logic (a sentence is sufficient).
|
@gfyoung Added comment. |
jreback
approved these changes
Nov 4, 2017
|
thanks @Licht-T |
gfyoung
added a commit
that referenced
this pull request
Nov 4, 2017
gfyoung
pushed a commit
that referenced
this pull request
Nov 4, 2017
(cherry picked from commit e0c9c6)
1kastner
pushed a commit
to 1kastner/pandas
that referenced
this pull request
Nov 5, 2017
1kastner
pushed a commit
to 1kastner/pandas
that referenced
this pull request
Nov 5, 2017
No-Stream
pushed a commit
to No-Stream/pandas
that referenced
this pull request
Nov 28, 2017
No-Stream
pushed a commit
to No-Stream/pandas
that referenced
this pull request
Nov 28, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
git diff upstream/master -u -- "*.py" | flake8 --diff