Skip to content

Chunksize from json memory consumption as high as without chunksize #34548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
3 tasks done
hasnain2808 opened this issue Jun 3, 2020 · 2 comments · Fixed by #38293
Closed
3 tasks done

Chunksize from json memory consumption as high as without chunksize #34548

hasnain2808 opened this issue Jun 3, 2020 · 2 comments · Fixed by #38293
Labels
IO JSON read_json, to_json, json_normalize Regression Functionality that used to work in a prior pandas version
Milestone
1.2

Comments

@hasnain2808
Copy link
Contributor

hasnain2808 commented Jun 3, 2020 •
edited
Loading

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.

While working on the PR (#33962) for adding nrows to read_json it was seen that maybe the chunksize parameter for read_json is not doing its task. ASV benchmarks were carried out but as this issue is not related to the PR we decided to create a new issue for it

Here is the link to the paste that has all the results
paste
The results are nicely added this to this spreadsheet
spreadsheet
@hasnain2808 hasnain2808 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 3, 2020
@hasnain2808
Copy link
Contributor Author

Someone please add the 'Needs Discussion' label

@hasnain2808 hasnain2808 changed the title BUG: Chunksize from json memory consumption as high as without chunksize Jun 3, 2020
@jbrockmendel jbrockmendel added IO JSON read_json, to_json, json_normalize and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 6, 2020
@fabriceyhc
Copy link

I'm having the same issue with pandas==1.1.0 and downgraded to pandas==1.0.0 in order to get a usable JsonReader.

@robertwb robertwb mentioned this issue Dec 4, 2020
Closed
2 tasks
robertwb added a commit to robertwb/pandas that referenced this issue Dec 4, 2020
@robertwb
Fix pandas-dev#36791 and pandas-dev#34548 correctly respecting read_j…
0b477a6 
…son chunksize.
@robertwb robertwb mentioned this issue Dec 4, 2020
Merged
6 tasks
@jreback jreback added this to the 1.2 milestone Dec 8, 2020
@jorisvandenbossche jorisvandenbossche added Regression Functionality that used to work in a prior pandas version and removed Bug labels Dec 11, 2020
@jreback jreback modified the milestones: 1.2, Contributions Welcome Dec 13, 2020
@jreback jreback modified the milestones: Contributions Welcome, 1.2 Dec 22, 2020
@simonm3 simonm3 mentioned this issue Jun 9, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
IO JSON read_json, to_json, json_normalize Regression Functionality that used to work in a prior pandas version
Projects
None yet
Milestone
1.2
Development

Successfully merging a pull request may close this issue.

BUG: read_json does not respect chunksize robertwb/pandas
5 participants
@jreback @jorisvandenbossche @fabriceyhc @jbrockmendel @hasnain2808