Skip to content

BUG: maybe_convert_objects mixed datetimes and timedeltas #27438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 24, 2019
Merged

BUG: maybe_convert_objects mixed datetimes and timedeltas #27438

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9f0c986
BUG: maybe_convert_objects mixed datetimes and timedeltas in some inp…
BeforeFlight Jul 17, 2019
f005531
Rewrite nested ifs. Test added.
BeforeFlight Jul 20, 2019
e109dbf
Some fixes.
BeforeFlight Jul 20, 2019
034f729
Tests with np.nan, pd.NaT added.
BeforeFlight Jul 20, 2019
d416771
Linter fixes.
BeforeFlight Jul 20, 2019
f2f265e
Array full of NaT ambiguity TODO.
BeforeFlight Jul 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 25 additions & 8 deletions pandas/_libs/lib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -942,6 +942,7 @@ cdef class Seen:

cdef:
bint int_ # seen_int
bint nat_ # seen nat
bint bool_ # seen_bool
bint null_ # seen_null
bint uint_ # seen_uint (unsigned integer)
Expand All @@ -965,6 +966,7 @@ cdef class Seen:
initial methods to convert to numeric fail.
"""
self.int_ = 0
self.nat_ = 0
self.bool_ = 0
self.null_ = 0
self.uint_ = 0
Expand Down Expand Up @@ -1044,11 +1046,13 @@ cdef class Seen:

@property
def is_bool(self):
return not (self.datetime_ or self.numeric_ or self.timedelta_)
return not (self.datetime_ or self.numeric_ or self.timedelta_
or self.nat_)

@property
def is_float_or_complex(self):
return not (self.bool_ or self.datetime_ or self.timedelta_)
return not (self.bool_ or self.datetime_ or self.timedelta_
or self.nat_)


cdef _try_infer_map(v):
Expand Down Expand Up @@ -1947,12 +1951,11 @@ def maybe_convert_objects(ndarray[object] objects, bint try_float=0,
seen.null_ = 1
floats[i] = complexes[i] = fnan
elif val is NaT:
seen.nat_ = 1
if convert_datetime:
idatetimes[i] = NPY_NAT
seen.datetime_ = 1
if convert_timedelta:
itimedeltas[i] = NPY_NAT
seen.timedelta_ = 1
if not (convert_datetime or convert_timedelta):
seen.object_ = 1
break
Expand Down Expand Up @@ -2046,11 +2049,18 @@ def maybe_convert_objects(ndarray[object] objects, bint try_float=0,
else:
if not seen.bool_:
if seen.datetime_:
if not seen.numeric_:
return datetimes
if not seen.timedelta_:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make this 1 level higher elif's, its very hard to follow these branches currently

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. As for me - this nested "if"s are ugly even before I've changed anything. Besides it is one of the good places to make a mistake (as it happens in current case).

if not seen.numeric_:
return datetimes
elif seen.timedelta_:
if not seen.numeric_:
return timedeltas
elif seen.nat_:
if not seen.numeric_:
if convert_datetime:
return datetimes
elif convert_timedelta:
return timedeltas
else:
if seen.complex_:
return complexes
Expand All @@ -2077,11 +2087,18 @@ def maybe_convert_objects(ndarray[object] objects, bint try_float=0,
else:
if not seen.bool_:
if seen.datetime_:
if not seen.numeric_:
return datetimes
if not seen.timedelta_:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Contributor Author

@BeforeFlight BeforeFlight Jul 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback I've proposed simple test in corresponding issue that returns undesired output - this PR is aimed to resolve exactly such situations. Should I add this single test only? If not - maybe_convert_objects need full test suite (for all possible types combinations) but as I tried to explain earlier there will be thousands of it which is not doable 'by hands'.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have quite a few tests
i am ok with systematic testing but don’t go overboard

so happy to merge tests first

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so happy to merge tests first

@jreback sorry but from your answer I still can't understand - is single test which is fixed by this PR enough or not?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this needs at least a single test

if u want to do more comprehensive tests that xfail some (that re not passing now) is ok too (can be before or after this PR)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add.

The problem with comprehensive tests: it cannot be done by hands -> it needs some "trusted" function to tested against i.e. needs copying old function for tests purposes only (as one of the options) - and here I don't understand may I do so or not.

if not seen.numeric_:
return datetimes
elif seen.timedelta_:
if not seen.numeric_:
return timedeltas
elif seen.nat_:
if not seen.numeric_:
if convert_datetime:
return datetimes
elif convert_timedelta:
return timedeltas
else:
if seen.complex_:
if not seen.int_:
Expand Down