Fixes #45506 Catch overflow error when converting to datetime

[Compro-Prasad]

Exception is raised when date part (like day, month, year) is greater than 32 bit signed integer

• xref BUG: Inconsistent equal comparison for datetime and string #45506
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[pep8speaks]

Hello @Compro-Prasad! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-01-31 14:17:45 UTC

[mroeschke]

Could you add a unit test?

Also I don't think this closes the original issue. I think in the original issue the strings should be just compared as strings not datetime-like objects.

[jeff-hernandez]

Yeah I was hoping the fix would result in a boolean series.

import pandas as pd

data = [
    {"f_0": "2015-07-01", "f_2": "08335394550"},
    {"f_0": "2015-07-02", "f_2": "+49 (0) 0345 300033"},
    {"f_0": "2015-07-03", "f_2": "+49(0)2598 04457"},
    {"f_0": "2015-07-04", "f_2": "0741470003"},
    {"f_0": "2015-07-05", "f_2": "04181 83668"},
]

dtypes = dict(f_0='datetime64[ns]', f_2='string')
df = pd.DataFrame(data=data).astype(dtypes)
df['f_0'].eq(df['f_2'])

0    False
1    False
2    False
3    False
4    False
dtype: bool

[jreback]

pls always start with a test that demonstrates the issue (from the OP)

[Compro-Prasad]

Could you add a unit test?

Sure. On it.

Also I don't think this closes the original issue. I think in the original issue the strings should be just compared as strings not datetime-like objects.

Guessing datatypes isn't the most pleasant things to work with. For efficiency its nice to convert both into strings. But if one is already a datetime like object then its more convenient to prefer datetime over string. If that fails, string should be preferred. This case from OP is not a real scenario anyway.

Yeah I was hoping the fix would result in a boolean series.

That is what I get after applying the fix:

Python 3.10.1 (main, Dec 18 2021, 23:53:45) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> 
>>> data = [
...     {"f_0": "2015-07-01", "f_2": "08335394550"},
...     {"f_0": "2015-07-02", "f_2": "+49 (0) 0345 300033"},
...     {"f_0": "2015-07-03", "f_2": "+49(0)2598 04457"},
...     {"f_0": "2015-07-04", "f_2": "0741470003"},
...     {"f_0": "2015-07-05", "f_2": "04181 83668"},
... ]
>>> 
>>> dtypes = dict(f_0='datetime64[ns]', f_2='string')
>>> df = pd.DataFrame(data=data).astype(dtypes)
>>> df['f_0'].eq(df['f_2'])
0    False
1    False
2    False
3    False
4    False
dtype: bool

pls always start with a test that demonstrates the issue (from the OP)

Sure thing. I am very new to pandas development. Thanks for the suggestion. I will add a test.

[Compro-Prasad]

Have added a test in pandas/tests/arrays/test_array.py. Is this the correct place where the test should go?

[Compro-Prasad]

I found the following error from mypy:

pandas/io/parquet.py:184: error: "RawIOBase" has no attribute "name"  [attr-defined]

IMO using .name instead of .raw.name should be a good enough solution.

Shall I make the changes?

[jreback]

I found the following error from mypy:

pandas/io/parquet.py:184: error: "RawIOBase" has no attribute "name"  [attr-defined]

IMO using .name instead of .raw.name should be a good enough solution.

Shall I make the changes?

ignore as this is taken care of elsewhere

[Compro-Prasad]

@jreback Have made the changes.

[jreback]

great, can you add a whatsnew entry in bug fixes for 1.5 (conversion or the datetime section is fine). and ping on green.

[Compro-Prasad]

@jreback Have added the whatsnew entry.

[jreback]

thanks @Compro-Prasad