Skip to content

ENH: Add dict support for read_csv date_format #51418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 16, 2023

Conversation

phofl
Copy link
Member

@phofl phofl commented Feb 15, 2023

@MarcoGorelli MarcoGorelli added Datetime Datetime data dtype IO Data IO issues that don't fit into a more specific label labels Feb 16, 2023
@MarcoGorelli MarcoGorelli added this to the 2.0 milestone Feb 16, 2023
@mroeschke mroeschke merged commit 7598f71 into pandas-dev:main Feb 16, 2023
@mroeschke
Copy link
Member

Thanks @phofl

@phofl phofl deleted the read_csv_date_format branch February 16, 2023 16:58
@phofl
Copy link
Member Author

phofl commented Feb 16, 2023

thx @MarcoGorelli for helping out

@mroeschke
Copy link
Member

So it appears the tests added here can raise a UserWarning

============================================================================= warnings summary =============================================================================
pandas/tests/io/parser/test_parse_dates.py::test_parse_dates_dict_format_two_columns[c_high-foo-parse_dates1]
pandas/tests/io/parser/test_parse_dates.py::test_parse_dates_dict_format_two_columns[c_low-foo-parse_dates1]
pandas/tests/io/parser/test_parse_dates.py::test_parse_dates_dict_format_two_columns[python-foo-parse_dates1]
  /.../opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'pandas._libs.tslibs.parsing._maybe_warn_about_dayfirst'

  Traceback (most recent call last):
    File ".../pandas/core/tools/datetimes.py", line 137, in _guess_datetime_format_for_array
      guessed_format = guess_datetime_format(
  UserWarning: Parsing dates in %d- %m-%Y format when dayfirst=False (the default) was specified. Pass `dayfirst=True` or specify a format to silence this warning.

    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

I would expect no warning since an explicit format is passed?

@MarcoGorelli
Copy link
Member

MarcoGorelli commented Feb 18, 2023

That's a good question - so, parse_dates is {'foo': [0, 1]}, and date_format specifies column 'foo', but in

date_fmt = (
date_format.get(col) if isinstance(date_format, dict) else date_format
)

, col is 'a_b', and so it doesn't find the date format in the dict and so date_fmt is None, so the format has to be guessed. It's reassuring that the correct format was guessed 😄

Thanks for spotting this

@phofl should the user have passed date_format={'a_b': '%d- %m-%Y'}, or does this need fixing in the code?

@phofl
Copy link
Member Author

phofl commented Feb 18, 2023

code I'd say, good catch btw.

@phofl
Copy link
Member Author

phofl commented Feb 18, 2023

#51479

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Datetime Datetime data dtype IO Data IO issues that don't fit into a more specific label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: let date_format take a mapping of columns to formats
3 participants