BUG: Cast pd.NA to pd.NaT in to_datetime

[dsaxton]

• closes pandas.to_datetime raises when given pd.NA #32213
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jorisvandenbossche]

I am not fully sure we should do this, since pd.NA and pd.NaT behave differently (and we probably want to have a timestamp dtype that uses pd.NA at some point)

[dsaxton]

I am not fully sure we should do this, since pd.NA and pd.NaT behave differently (and we probably want to have a timestamp dtype that uses pd.NA at some point)

Ok, I'll close the PR, but should probably leave the issue open for now?

[jbrockmendel]

if the user specifically calls to_datetime, it makes sense that pd.NA would cast to NaT

[jorisvandenbossche]

@jbrockmendel so the reason I initially didn't add this check to checknull_with_nat is because this adds a dependency of tslibs on _libs/missing.pyx, while now the dependency is actually the other way around.

We might want to make missing.pyx independent on the rest of _libs, so tslibs can easily depend on it.

[jreback]

@jbrockmendel is this a problem? (the dependency)?

[jbrockmendel]

I'd like to avoid it if possible. The property "tslibs doesnt rely on anything except _config" is really nice for reasoning-about-code purposes.

@dsaxton is it feasible to put pd.NA into a higher-level check than checknull_with_nat?

[dsaxton]

Hmm, it may be but I think it would have to be before https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/datetimes.py#L1844, do you think that looks possible? There's also the "trick" of not importing NA at all and checking for (val == val) is (val != val) in checknull_with_nat

[jreback]

ok, going to merge this as-is. @jbrockmendel or @dsaxton if you can figure a way to decouple this a bit would be great (though since missing is pretty dependency free i actually don't mind this)

[jbrockmendel]

@dsaxton im having trouble figuring out which usage of checknull_with_nat is the relevant one here, can you point it out? The only ones I see when grepping are in libmissing.checknull (which already checks for C_NA, which this renders redundant), and in Period.__new__, which seems like it shouldn't be relevant.

[jbrockmendel]

looks like i was confusing checknull_with_nat with is_null_datetimelike, never mind

[jbrockmendel]

(this suggests we may have too many null-checking-like functions)

[jbrockmendel]

We might want to make missing.pyx independent on the rest of _libs, so tslibs can easily depend on it.

Maybe a file with just NAType, as that doesn't depend on most of the things in libmissing (many of which do depend on tslibs). This seems like it will be necessary if we want e.g. pd.Timestamp(pd.NA) to give pd.NaT instead of raising

[jbrockmendel]

Is this something we need for 1.0.2, or can it wait for 1.1.0? if the latter id like to revert and take more time to figure out the dependency structure

[jreback]

Is this something we need for 1.0.2, or can it wait for 1.1.0? if the latter id like to revert and take more time to figure out the dependency structure

this is a usability issue in 1.0.2

happy to change for 1.1 to a proper structure (this is already merged and backport ed)