Skip to content

BUG: Fix precise_xstrtod segfault on long exponent #38789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Dec 30, 2020

Conversation

mzeitlin11
Copy link
Member

@snowman2 can you check if this branch returns behavior to what you had before 1.2?

@mzeitlin11 mzeitlin11 added Regression Functionality that used to work in a prior pandas version IO CSV read_csv, to_csv Segfault Non-Recoverable Error labels Dec 29, 2020
# GH#38753
parser = all_parsers
data = f"data\n10E{neg_exp}"
for precision in parser.float_precision_choices:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can parameterize on these too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue I ran into is that the parsers don't have consistent potential float_precisions, eg PythonParser only has None. Could parameterize and just assert failure on disallowed combinations of float_precision and parser, but that seems ugly. Will try to think of a better solution.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some kind of fixture-within-a-fixture functionality would be nice, though not sure if that's feasible...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a new fixture, all_parsers_all_precisions. Not the most elegant solution, essentially just precomputes all allowed combos and uses that as the source for the fixture. If you have a better idea please let me know!

# GH#38753
parser = all_parsers
data = f"data\n10E{exp}"
for precision in parser.float_precision_choices:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

@jreback jreback added this to the 1.2.1 milestone Dec 29, 2020
@jreback
Copy link
Contributor

jreback commented Dec 29, 2020

cc @gfyoung

@snowman2
Copy link

can you check if this branch returns behavior to what you had before 1.2?

pandas.read_csv successfully read in the CSV file with this branch without issue.

if precision == "round_trip":
if exp == 999999999999999999 and is_platform_linux():
mark = pytest.mark.xfail(reason="On Linux gives object result")
request.node.add_marker(mark)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This discrepancy is unrelated, occurs on master, have opened #38794 for it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk can you put the issue number in the xfail itself.

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small comment, ping on green.

if precision == "round_trip":
if exp == 999999999999999999 and is_platform_linux():
mark = pytest.mark.xfail(reason="On Linux gives object result")
request.node.add_marker(mark)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk can you put the issue number in the xfail itself.

@jreback jreback merged commit b438f74 into pandas-dev:master Dec 30, 2020
@jreback
Copy link
Contributor

jreback commented Dec 30, 2020

thanks @mzeitlin11 very nice, keep em coming!

@jreback
Copy link
Contributor

jreback commented Dec 30, 2020

@meeseeksdev backport 1.2.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
IO CSV read_csv, to_csv Regression Functionality that used to work in a prior pandas version Segfault Non-Recoverable Error
Projects
None yet
Development

Successfully merging this pull request may close these issues.

REGR: pd.read_csv segfaults with 1.2 (has worked since before pandas 1.0)
4 participants