BUG: Improve error message for skipfooter malformed rows in Python engine #14749
Conversation
jreback
added
Error Reporting
| if self.skipfooter > 0: | ||
| reason = ('Error could possibly be due to ' | ||
| 'skipfooter being ignored in Python\'s ' | ||
| 'native csv library.') |
There was a problem hiding this comment.
I am wondering if we can word this a little different. As now it seems to imply that skipfooter is ignored altogether ("so why can I specify it then").
Do I understand it correctly that the skipped lines are still parsed, but not used when converting the parsed lines to a DataFrame (so eg the number of values in the skipped lines do not need to match the rest of the file)?
There was a problem hiding this comment.
Yes, Python's native CSV library does not care about skipfooter, so any malformed lines in the skipfooter section will raise an error. However, if there are no such errors, these rows are then discarded afterwards.
There was a problem hiding this comment.
@jorisvandenbossche : Any thoughts on rewording (if any)?
There was a problem hiding this comment.
Trying a long version:
Error could possibly be due to parsing errors in the skipped footer rows (the
skipfooterkeyword is only applied after Python's csv library has parsed all rows)
(is that correct?)
If you have ideas to shorten it a bit but keep it clear, welcome :-)
There was a problem hiding this comment.
I like your suggestion! I'll change it to that.
Current coverage is 85.27% (diff: 100%)@@ master #14749 diff @@
==========================================
Files 144 144
Lines 50911 50915 +4
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 43413 43419 +6
+ Misses 7498 7496 -2
Partials 0 0
|
gfyoung
force-pushed
the
skipfooter-python-error
branch
2 times, most recently
from
8bcfb77 to
9b1d065
Compare
…gine Python's native CSV library does not respect the skipfooter parameter, so if one of those skipped rows is malformed, it will still raise an error. Closes pandas-devgh-13879.
gfyoung
force-pushed
the
skipfooter-python-error
branch
from
9b1d065 to
8aae4fe
Compare
|
@jreback , @jorisvandenbossche : Updated the error message, and everything is happy. Ready to merge if there are no other concerns. |
|
@gfyoung Thanks! |
Python's native CSV library does not respect the
skipfooterparameter, so if one of those skippedrows is malformed, it will still raise an error. Append some useful information to the error message regarding that if that situation is potentially applicable.
Closes #13879.