BUG: to_datetime behaves differently depending of the format of the string provided

[apj-graham]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Python 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.to_datetime("2016-03-11", dayfirst=True)
Timestamp('2016-11-03 00:00:00')
>>> pd.to_datetime("20160311", dayfirst=True)
Timestamp('2016-03-11 00:00:00')
>>>

Issue Description

Dates of the format "YYYYDDMM" are not converted correctly by to_datetime() when dayfirst=True. to_datetime() still reads the date characters as the months

Expected Behavior

Python 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import pandas as pd
pd.to_datetime("2016-03-11", dayfirst=True)
Timestamp('2016-11-03 00:00:00')
pd.to_datetime("20160311", dayfirst=True)
Timestamp('2016-11-03 00:00:00')

Installed Versions

INSTALLED VERSIONS

commit : a671b5a
python : 3.11.7.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-1160.114.2.el7.x86_64
Version : #1 SMP Sun Mar 3 08:18:39 EST 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.4
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 23.3.2
Cython : 3.0.7
pytest : 7.4.4
hypothesis : 6.94.0
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : 5.1.0
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.20.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : 2023.10.1
fsspec : 2023.12.2
gcsfs : None
matplotlib : 3.8.2
numba : 0.58.1
numexpr : 2.8.8
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 16.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.4
sqlalchemy : 2.0.25
tables : 3.9.2
tabulate : 0.9.0
xarray : 2023.7.0
xlrd : 2.0.1
zstandard : 0.19.0
tzdata : 2023.4
qtpy : 2.4.1
pyqt5 : None

[Aloqeely]

Thanks for the report! pandas could not guess the format of the second string so it relied on the dateutil library for parsing it which does not take the dayfirst parameter in consideration. It's generally best to provide the format parameter for when you're using unusual date formats.

I suppose we should at least warn for when dayfirst is provided and the date format wasn't guessed (cc @MarcoGorelli)

[MarcoGorelli]

thanks @apj-graham for the report

It's not dateutil, it's pandas' own parser here, looks like dayfirst isn't getting passed to

pandas/pandas/_libs/tslibs/conversion.pyx

Lines 609 to 612 in cbe1b32

	string_to_dts_failed = string_to_dts(
	ts, &dts, &out_bestunit, &out_local,
	&out_tzoffset, False
	)

And it probably shouldn't - that path should only be used for iso8601-like strings

I think the fix here is to not use that path if dayfirst was specified - @Aloqeely do you want to repurpose your PR to do that?

[Aloqeely]

Thanks for the correction.

do you want to repurpose your PR to do that?

Alright, I'll take a look