BUG: Bug in DataFrame construction with nulls and datetimes in a list like

[jreback]

closes #15869

[chrisaycock]

Is seen_string supposed to be used somewhere?

[jreback]

actually this is non-performant.

The issue is that it ends up looking at all of the strings, when we need to just test a couple to see if they are convertible. Not to mention this fails a couple of tests that have to do with us preserving date (which is annoying).

[chrisaycock]

Similarly, is seen_null supposed to be used somewhere?

[jreback]

no these were tries.......

[jreback]

I think this is fixed. boy user convenience is a PITA.

[jreback]

ok I think its fixed now.

[chrisaycock]

As long as the test/benchmarks pass, then LGTM. This was a bigger effort than I thought it would be.

[jreback]

@chrisaycock yeah a lot of this lower-level inference code is reasonable as we have quite a few integration tests, so refactoring stuff that I wrote years ago is pretty straightforward. But of course there are lots of cases and easy to break stuff (which the tests catch).

Down the road we will will be able to simplify things. For example datetime.date is not a first class type, so its very hacky when we infer from it (to maintain some compat). In the future, it will be more straightforward to do this (as we will have a valid first class type).

Further I have to think about whether we still want things like:

In [5]: Series([pd.NaT, '1 day'])
Out[5]: 
0      NaT
1   1 days
dtype: timedelta64[ns]

to work. It is reasonable & not too hard.

this is actually a bigger issue:

In [6]: Series([pd.NaT])
Out[6]: 
0   NaT
dtype: datetime64[ns]

as this is somewhat arbitrary. but this is for another time.

[codecov]

Codecov Report

Merging #15892 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #15892      +/-   ##
==========================================
- Coverage   90.97%   90.97%   -0.01%     
==========================================
  Files         145      145              
  Lines       49491    49495       +4     
==========================================
+ Hits        45023    45026       +3     
- Misses       4468     4469       +1

Flag	Coverage Δ
#multiple	88.73% <100%> (-0.01%)	⬇️
#single	40.63% <70%> (-0.01%)	⬇️

Impacted Files	Coverage Δ
pandas/types/cast.py	85.71% <100%> (+0.11%)	⬆️
pandas/core/common.py	90.68% <0%> (-0.35%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e50d397...6bf2148. Read the comment docs.