Skip to content

BUG: read_csv with custom date parser and na_filter=True results in ValueError #39079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 16 commits into from
Closed

BUG: read_csv with custom date parser and na_filter=True results in ValueError #39079

wants to merge 16 commits into from

Conversation

ftrihardjo
Copy link
Contributor

@ftrihardjo ftrihardjo commented Jan 10, 2021 •
edited by jreback
Loading

@pep8speaks
Copy link

pep8speaks commented Jan 10, 2021 •
edited
Loading

Hello @ftrihardjo! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 115:1: W293 blank line contains whitespace
Line 116:5: E128 continuation line under-indented for visual indent
Line 117:9: E128 continuation line under-indented for visual indent
Line 118:9: E128 continuation line under-indented for visual indent
Line 118:18: E251 unexpected spaces around keyword / parameter equals
Line 118:20: E251 unexpected spaces around keyword / parameter equals
Line 119:9: E128 continuation line under-indented for visual indent
Line 119:15: E251 unexpected spaces around keyword / parameter equals
Line 119:17: E251 unexpected spaces around keyword / parameter equals
Line 120:28: E251 unexpected spaces around keyword / parameter equals
Line 120:30: E251 unexpected spaces around keyword / parameter equals
Line 121:17: E251 unexpected spaces around keyword / parameter equals
Line 121:19: E251 unexpected spaces around keyword / parameter equals
Line 123:15: E251 unexpected spaces around keyword / parameter equals
Line 123:17: E251 unexpected spaces around keyword / parameter equals
Line 124:27: E251 unexpected spaces around keyword / parameter equals
Line 124:29: E251 unexpected spaces around keyword / parameter equals
Line 125:28: E251 unexpected spaces around keyword / parameter equals
Line 125:30: E251 unexpected spaces around keyword / parameter equals
Line 126:15: E251 unexpected spaces around keyword / parameter equals
Line 126:17: E251 unexpected spaces around keyword / parameter equals

Comment last updated at 2021-01-19 21:48:07 UTC

@jreback jreback changed the title pandas-dev issue #36111 BUG: read_csv with custom date parser and na_filter=True results in ValueError Jan 10, 2021
@jreback jreback added IO CSV read_csv, to_csv Testing pandas testing functions or related to the test suite labels Jan 10, 2021
jreback
jreback requested changes Jan 10, 2021
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to compare the result with an explictly constructed frame, see other tests for examples

pandas/tests/io/test_common.py Outdated
@@ -139,6 +139,22 @@ def test_iterator(self):
tm.assert_frame_equal(first, expected.iloc[[0]])
tm.assert_frame_equal(pd.concat(it), expected.iloc[1:])

def test_read_csv_with_custom_date_parser(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move to pandas/tests/io/parser//test_parse_dates.py

pandas/tests/io/test_common.py Outdated
41051.00 -98573.7302 871458.0640 389.0086
""")

df = pd.read_csv(testdata, delim_whitespace=True, parse_dates=True, date_parser=__custom_date_parser, index_col='time')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we want to test with the fixture all_parsers

pandas/tests/io/test_common.py Outdated
def test_read_csv_with_custom_date_parser(self):
# GH36111
def __custom_date_parser(time):
time_temp = time.astype(np.float).astype(np.int) # convert float seconds to int type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this won't pass linting

@jreback jreback added the Timedelta Timedelta data type label Jan 10, 2021
jreback
jreback requested changes Jan 13, 2021
View reviewed changes
pandas/tests/io/parser/test_parse_dates.py Outdated
@@ -55,6 +55,39 @@ def test_separator_date_conflict(all_parsers):
)
tm.assert_frame_equal(df, expected)

def test_read_csv_with_custom_date_parser(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the allparser fixture (and thus parser.read_csv) see the test above for how

jreback
jreback requested changes Jan 14, 2021
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you also have some liniting errors, pls use precommit locally to fix

pandas/tests/io/parser/test_parse_dates.py Outdated
@@ -55,6 +55,40 @@ def test_separator_date_conflict(all_parsers):
)
tm.assert_frame_equal(df, expected)

@pytest.fixtures(all_parsers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need this line at all

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how to run pre-commit for a single file because running pre-commit run --all-files like in pre-commit website is too slow for my computer

@jreback jreback added this to the 1.3 milestone Jan 15, 2021
@ftrihardjo ftrihardjo closed this Jan 31, 2021
@ftrihardjo
Copy link
Contributor Author

pandas-dev issue #36111 #39496

@simonjayhawkins simonjayhawkins mentioned this pull request Feb 1, 2021
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback requested changes

Assignees
No one assigned
Labels
IO CSV read_csv, to_csv Testing pandas testing functions or related to the test suite Timedelta Timedelta data type
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.

BUG: read_csv with custom date parser and na_filter=True results in ValueError
3 participants
@ftrihardjo @pep8speaks @jreback