Skip to content

BUG: parse_dates=False while passing date_parser tries to use date parser #44599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Nov 25, 2021
Merged

BUG: parse_dates=False while passing date_parser tries to use date parser #44599

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cbbf7c1
BUG: Providing fix for GH#44366
loicdiridollou Nov 24, 2021
84f8f63
BUG: fix pre-commit run
loicdiridollou Nov 24, 2021
4776854
BUG: adding other API changes for change of default argument
loicdiridollou Nov 24, 2021
c5eec17
Rewriting if statement and adding test
loicdiridollou Nov 24, 2021
465a6f1
BUG: changing pd.Series to Series to validate pre-commit hook
loicdiridollou Nov 24, 2021
3205754
Removing the second whatsnew statement only to keep the bug fix one
loicdiridollou Nov 24, 2021
a69094b
BUG: simplifying the test with parser and parse_dates False #44366
loicdiridollou Nov 24, 2021
56e68cf
BUG: fixing pre-commit #44366
loicdiridollou Nov 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v1.4.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,7 +400,7 @@ Other API changes
- :meth:`Index.get_indexer_for` no longer accepts keyword arguments (other than 'target'); in the past these would be silently ignored if the index was not unique (:issue:`42310`)
- Change in the position of the ``min_rows`` argument in :meth:`DataFrame.to_string` due to change in the docstring (:issue:`44304`)
- Reduction operations for :class:`DataFrame` or :class:`Series` now raising a ``ValueError`` when ``None`` is passed for ``skipna`` (:issue:`44178`)
-
- Change in the default argument for ``parse_dates`` from ``False`` to ``None`` in :func:`read_csv` (:issue:`44366`)

.. ---------------------------------------------------------------------------

Expand Down Expand Up @@ -653,6 +653,7 @@ I/O
- Bug in :func:`read_csv` raising ``ValueError`` when ``parse_dates`` was used with ``MultiIndex`` columns (:issue:`8991`)
- Bug in :func:`read_csv` raising ``AttributeError`` when attempting to read a .csv file and infer index column dtype from an nullable integer type (:issue:`44079`)
- :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` with ``compression`` set to ``'zip'`` no longer create a zip file containing a file ending with ".zip". Instead, they try to infer the inner file name more smartly. (:issue:`39465`)
- Bug in :func:`read_csv` when passing simultaneously a parser in ``date_parser`` and ``parse_dates=False``, the parsing was still called (:issue:`44366`)

Period
^^^^^^
Expand Down
10 changes: 8 additions & 2 deletions pandas/io/parsers/readers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,8 +511,14 @@ def _read(
):
"""Generic reader of line files."""
if kwds.get("date_parser", None) is not None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to read. Please do it somehow like this:

if kwds.get("date_parser", None) is not None and kwds.get("pars_dates", None) is None:
    kwds.get["pars_dates"] = True
elif kwds.get("pars_dates", None) is None:
    kwds.get["pars_dates"] = False

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct! I have made the changes so now it is more readable, added a test to see if the parser is not called when we set parse_dates to False.
Some tests are failing for an obscure reason as they are not related to the code change (and they are also failing on other builds)

# if we pass a date_parser and parse_dates=False, we should not parse the
# dates GH#44366
if isinstance(kwds["parse_dates"], bool):
kwds["parse_dates"] = True
pass
else:
kwds["parse_dates"] = (
True if kwds["parse_dates"] is None else kwds["parse_dates"]
)

# Extract some of the arguments (pass chunksize on).
iterator = kwds.get("iterator", False)
Expand Down Expand Up @@ -585,7 +591,7 @@ def read_csv(
verbose=False,
skip_blank_lines=True,
# Datetime Handling
parse_dates=False,
parse_dates=None,
infer_datetime_format=False,
keep_date_col=False,
date_parser=None,
Expand Down