TST: Parse dates with empty space (#6428) #14862
Conversation
Current coverage is 85.30% (diff: 100%)@@ master #14862 diff @@
==========================================
Files 144 144
Lines 50957 50957
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
Hits 43469 43469
Misses 7488 7488
Partials 0 0
|
| def test_to_datetime_with_space_in_series(self): | ||
| # GH 6428 | ||
| s = Series(['10/18/2006', '10/18/2008', ' ']) | ||
| tm.assertRaises(ValueError, lambda: to_datetime(s, errors='raise')) |
| # GH 6428 | ||
| data = """case,opdate | ||
| 7,10/18/2006 | ||
| 7,10/18/2008 |
There was a problem hiding this comment.
I think this should actually parse as datetime and make the space a NaT. So this 'works' in that it doesn't convert as today's date, but it is not registering as missing here.
There was a problem hiding this comment.
Currently it looks like errors='ignore' is set when parsing datetimes with read_csv (example). I can change this to errors='coerce'
There was a problem hiding this comment.
hmm, I think we did that on purpose..
that whole clause is bogus, errors='ignore' means it will never raise.
but its a bit tricky, you want to try to parse, then fallback on a softer-parse if necessary.
try playing around with this and see what happens.
There was a problem hiding this comment.
Ah thanks for the insight. I'll tinker with it later tonight.
There was a problem hiding this comment.
You were correct; current tests require errors='ignore'.
This is tricky since errors='ignore' and errors='coerce' both terminate the parse with NaTs or returning the input. Would it be possible to introduce a new kwarg (e.g. date_errors) to pass ignore/coerce/raise, into to_datetime within read_csv? It could default to ignore.
There was a problem hiding this comment.
Would it be possible to introduce a new kwarg (e.g. date_errors) to pass ignore/coerce/raise, into to_datetime within read_csv? It could default to ignore.
This would make the API even more complicated. Generally supporting non-standard datetime parsing should simply use pd.to_datetime post-read_csv.
@jorisvandenbossche any thoughts here.
There was a problem hiding this comment.
Fair point.
I could clarify in the docs that if parse_dates cannot parse a date the column is returned untouched (which seems like the current behavior) and the user can coerce a space or any unparseable date to NaT post-read_csv with to_datetime
There was a problem hiding this comment.
yeah that's prob the best soln here (pls add to doc-string & a small warning/note section in io.rst if you don't mind).
There was a problem hiding this comment.
yep, I agree that we should not add kwargs to read_csv for something like this. If you want the date parser, your dates should just be able to be parsed with the default settings, otherwise you can just use to_datetime. PR for doc clarification certainly welcome!
There was a problem hiding this comment.
PR for doc clarification certainly welcome!
which you already included here :-)
jreback
added
Bug
IO CSV
Add doc explaining parse_date limitation
|
@mroeschke Thanks! |
+ Add doc explaining parse_date limitation
git diff upstream/master | flake8 --diffParsing a date with an empty space no longer returns today's date on master 0.19.1. Not sure when this was fixed, but it doesn't seem like it occurred recently.