BUG: DataFrame.replace with out of bound datetime causing RecursionError #22108
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #22108 +/- ##
==========================================
+ Coverage 92.06% 92.07% +0.01%
==========================================
Files 170 170
Lines 50705 50693 -12
==========================================
- Hits 46680 46675 -5
+ Misses 4025 4018 -7
Continue to review full report at Codecov.
|
jreback
requested changes
Jul 29, 2018
pandas/core/dtypes/cast.py
Outdated
| values = lib.maybe_convert_objects(values, | ||
| convert_datetime=datetime) | ||
| except ValueError: | ||
| pass |
There was a problem hiding this comment.
can you add a comment on when the valueerror is hit
There was a problem hiding this comment.
commented, used OutOfBoundsDatetime to be more clear on the error
| datetime(2018, 7, 28), | ||
| datetime(2018, 5, 28)])}), | ||
| datetime(2018, 5, 28), datetime(2018, 7, 28), | ||
| DataFrame({'datetime64': Index([datetime(2018, 7, 28)] * 3)})), |
There was a problem hiding this comment.
can you add a datetime64 w/tz here as well (if its an easy add)
|
@jreback items actioned. additional feedback welcome. |
jreback
approved these changes
Aug 1, 2018
|
thanks @minggli keep me coming! |
|
Thanks for reviewing! |
dberenbaum
pushed a commit
to dberenbaum/pandas
that referenced
this pull request
Aug 3, 2018
Sup3rGeo
pushed a commit
to Sup3rGeo/pandas
that referenced
this pull request
Oct 1, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
git diff upstream/master -u -- "*.py" | flake8 --diffBy default, keyword convert=True for replace method for Block and ObjectBlock. When trying to replace dataframe, blocks are operated on separately, during which conversion happens if convert=True. OutOfBoundsDatetime(inherited from ValueError) raised by lib.maybe_convert_objects caused Block.replace, ObjectBlock.replace to form infinite recursive loop.
As mentioned in #20380, setting convert=False (private and not available to public) solves the issue at hand, but there appear to be other uses cases that expect it to be True by default. Right now, simply wrap a try...except block around the lib.maybe_convert_objects in cast module. Downside is it catches all ValueError in maybe_convert_objects silently.