BUG: parse_dates may have columns not in dataframe #32320

sathyz · 2020-02-28T03:28:41Z

read_csv will raise ValueError when columnes used for parse_dates are found in the dataframe.

closes read_csv: if parse_dates dont appear in use_cols, we get a trace #31251
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

read_csv will raise ValueError when columnes used for parse_dates are found in the dataframe.

MarcoGorelli

Thanks @sathyz

Just making some minor comments ahead of core members' review(s)

pandas/io/parsers.py

sathyz · 2020-02-28T16:36:35Z

Hi @jreback @gfyoung - Could you please review. I incorporated the changes given in #31815

doc/source/whatsnew/v1.1.0.rst

pandas/tests/io/parser/test_parse_dates.py

WillAyd

Thanks for the PR! A mix of minor things / comments

pandas/tests/io/parser/test_parse_dates.py

pandas/io/parsers.py

pep8speaks · 2020-02-29T03:46:40Z

Hello @sathyz! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-03-12 02:35:27 UTC

jreback

small comment, other lgtm.

doc/source/whatsnew/v1.1.0.rst

gfyoung · 2020-03-03T07:15:34Z

One minor doc fix, but otherwise, these changes look good. Nice job with the testing!

sathyz · 2020-03-04T14:59:05Z

@gfyoung done! Thanks for the corrections. I didn't notice these mistakes.

gfyoung · 2020-03-04T18:02:25Z

@pandas-dev/pandas-core : These mypy failures here look unrelated to this PR.

simonjayhawkins · 2020-03-04T18:33:52Z

@pandas-dev/pandas-core : These mypy failures here look unrelated to this PR.

xref #32438

sathyz · 2020-03-05T16:59:19Z

What do I have to do? Do I have to fix it to merge this PR?

simonjayhawkins · 2020-03-05T17:01:44Z

What do I have to do? Do I have to fix it to merge this PR?

you need to merge master see https://pandas.io/docs/development/contributing.html#updating-your-pull-request

sathyz · 2020-03-06T02:14:09Z

Last couple of times I did that #31815 #31550 and it was not smooth. Let me try this time.

sathyz · 2020-03-07T02:28:57Z

I am not sure what is failing in docs, how do I debug.

gfyoung · 2020-03-07T02:39:12Z

@pandas-dev/pandas-core

sathyz · 2020-03-10T14:15:03Z

Any updates? How do I fix the problem in docs?

datapythonista · 2020-03-10T14:32:08Z

Any updates? How do I fix the problem in docs?

The docs fetch a url from some other projects, to know the absolute url of a page when we write something like :ref:numpy.array. Looks like an error in GitHub pages was making the StatsModels website fail, and that caused the fetching of their url to fail in the build in your PR.

I guess that should be fixed not. A simple git fetch upstream && git merge upstream/master && git push should restart all jobs, and hopefully get the CI green this time.

pandas/io/parsers.py

sathyz · 2020-03-12T15:59:37Z

Done, please merge.

jreback

ping on green.

pandas/io/parsers.py

jreback · 2020-03-14T19:44:53Z

pandas/tests/io/parser/test_parse_dates.py

+@pytest.mark.parametrize(
+    "names, usecols, parse_dates, missing_cols",
+    [
+        (None, ["val"], ["date", "time"], "date, time"),


add a tuple or other list-like in this test

When I use tuple, _validate_parse_dates_arg throws the following error.

TypeError: Only booleans, lists, and dictionaries are accepted for the 'parse_dates' parameter

Yea I guess this is only currently documented as supporting scalars, lists and dicts (not tuples)

sathyz · 2020-03-16T16:27:50Z

@jreback done. Please review.

WillAyd · 2020-03-17T00:03:23Z

Thanks @sathyz - great first PR

BUG: parse_dates may have columns not in dataframe

d24b57a

read_csv will raise ValueError when columnes used for parse_dates are found in the dataframe.

MarcoGorelli suggested changes Feb 28, 2020

View reviewed changes

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

add return annotation.

3a99b39

jbrockmendel reviewed Feb 28, 2020

View reviewed changes

doc/source/whatsnew/v1.1.0.rst Outdated Show resolved Hide resolved

jbrockmendel reviewed Feb 28, 2020

View reviewed changes

pandas/tests/io/parser/test_parse_dates.py Outdated Show resolved Hide resolved

WillAyd requested changes Feb 29, 2020

View reviewed changes

pandas/tests/io/parser/test_parse_dates.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Show resolved Hide resolved

WillAyd added the IO CSV read_csv, to_csv label Feb 29, 2020

use chain.from_iterable to read parse_dates

78ff312

sathyz added 2 commits February 29, 2020 09:25

break long lines.

007c992

fixing typing mistake in cols_needed

7f1cd69

jreback requested changes Mar 3, 2020

View reviewed changes

doc/source/whatsnew/v1.1.0.rst Outdated Show resolved Hide resolved

jreback requested a review from gfyoung March 3, 2020 03:19

jreback added this to the 1.1 milestone Mar 3, 2020

jreback added the Error Reporting Incorrect or improved errors from pandas label Mar 3, 2020

jreback mentioned this pull request Mar 3, 2020

check parser_dates names in columns #31815

Closed

5 tasks

add func reference for read_csv in whatsnew entry

110f594

gfyoung reviewed Mar 3, 2020

View reviewed changes

doc/source/whatsnew/v1.1.0.rst Outdated Show resolved Hide resolved

docstring fix in whatsnew.

1536b77

Merge remote-tracking branch 'upstream/master' into issue-31251-2

6272a7d

datapythonista reviewed Mar 10, 2020

View reviewed changes

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

pandas/io/parsers.py Outdated Show resolved Hide resolved

sathyz added 5 commits March 11, 2020 10:13

Merge remote-tracking branch 'upstream/master' into issue-31251-2

57c114d

import itertools directly

ee4f3fb

typing hint for cols_needed

633e481

sort import statemeents

537f4df

Merge remote-tracking branch 'upstream/master' into issue-31251-2

22c399b

gfyoung approved these changes Mar 12, 2020

View reviewed changes

jreback requested changes Mar 14, 2020

View reviewed changes

use is_dict_like & is_list_like

337efcd

WillAyd approved these changes Mar 17, 2020

View reviewed changes

WillAyd merged commit 9c7494a into pandas-dev:master Mar 17, 2020

sathyz deleted the issue-31251-2 branch March 17, 2020 03:49

SeeminSyed pushed a commit to CSCD01-team01/pandas that referenced this pull request Mar 22, 2020

BUG: parse_dates may have columns not in dataframe (pandas-dev#32320)

e6848eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: parse_dates may have columns not in dataframe #32320

BUG: parse_dates may have columns not in dataframe #32320

sathyz commented Feb 28, 2020 •

edited

Loading

MarcoGorelli left a comment

sathyz commented Feb 28, 2020

WillAyd left a comment

pep8speaks commented Feb 29, 2020 •

edited

Loading

jreback left a comment

gfyoung commented Mar 3, 2020 •

edited

Loading

sathyz commented Mar 4, 2020

gfyoung commented Mar 4, 2020

simonjayhawkins commented Mar 4, 2020

sathyz commented Mar 5, 2020

simonjayhawkins commented Mar 5, 2020

sathyz commented Mar 6, 2020

sathyz commented Mar 7, 2020 •

edited

Loading

gfyoung commented Mar 7, 2020

sathyz commented Mar 10, 2020 •

edited

Loading

datapythonista commented Mar 10, 2020

sathyz commented Mar 12, 2020

jreback left a comment

jreback Mar 14, 2020

sathyz Mar 16, 2020

WillAyd Mar 17, 2020

sathyz commented Mar 16, 2020

WillAyd commented Mar 17, 2020

BUG: parse_dates may have columns not in dataframe #32320

BUG: parse_dates may have columns not in dataframe #32320

Conversation

sathyz commented Feb 28, 2020 • edited Loading

MarcoGorelli left a comment

Choose a reason for hiding this comment

sathyz commented Feb 28, 2020

WillAyd left a comment

Choose a reason for hiding this comment

pep8speaks commented Feb 29, 2020 • edited Loading

Comment last updated at 2020-03-12 02:35:27 UTC

jreback left a comment

Choose a reason for hiding this comment

gfyoung commented Mar 3, 2020 • edited Loading

sathyz commented Mar 4, 2020

gfyoung commented Mar 4, 2020

simonjayhawkins commented Mar 4, 2020

sathyz commented Mar 5, 2020

simonjayhawkins commented Mar 5, 2020

sathyz commented Mar 6, 2020

sathyz commented Mar 7, 2020 • edited Loading

gfyoung commented Mar 7, 2020

sathyz commented Mar 10, 2020 • edited Loading

datapythonista commented Mar 10, 2020

sathyz commented Mar 12, 2020

jreback left a comment

Choose a reason for hiding this comment

jreback Mar 14, 2020

Choose a reason for hiding this comment

sathyz Mar 16, 2020

Choose a reason for hiding this comment

WillAyd Mar 17, 2020

Choose a reason for hiding this comment

sathyz commented Mar 16, 2020

WillAyd commented Mar 17, 2020

sathyz commented Feb 28, 2020 •

edited

Loading

pep8speaks commented Feb 29, 2020 •

edited

Loading

gfyoung commented Mar 3, 2020 •

edited

Loading

sathyz commented Mar 7, 2020 •

edited

Loading

sathyz commented Mar 10, 2020 •

edited

Loading