Error in pd.to_datetime, with argument np.datetime64 array of 51 elements

[pajachiet]

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np
d = np.datetime64("2000-01-01", "s")

pd.to_datetime(np.full(50, d))
# Works

pd.to_datetime(np.full(51,  d))
# ERROR
# pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 29999249045-07-29 00:00:00



pd.to_datetime(pd.Series(np.full(51,  d)))

Problem description

With pandas 1.0, we get a new issue when using pd.to_datetime to convert an existing np.datetime64 array with 51 elements.

This works with 50 elements, or when converting to a Series beforehand, so I believe the problems comes from inside implicit conversion.

[jorisvandenbossche]

@pajachiet Thanks for the report!

[TomAugspurger]

The difference between 50 and 51 is from the caching we do in to_datetime

pandas/pandas/core/tools/datetimes.py

Lines 98 to 101 in ba08390

	if check_count is None:
	# in this case, the gain from caching is negligible
	if len(arg) <= 50:
	return False

[TomAugspurger]

The bug is in unique

In [20]: d = np.array([np.datetime64("2000-01-01", "s")])

In [21]: pd.unique(d)
Out[21]: array(['29999249045-07-29T00:00:00'], dtype='datetime64[s]')

So we should either handle non-ns datetimes there, or convert to ns before calling it.

[TomAugspurger]

Still working on this. Getting closer, but turned up a second issue with errors='coerce' and OOB timestamps.