BUG: read_json does not respect chunksize

[robertwb]

• closes How to use nrows along with chunksize in read_json() #36791
• closes Chunksize from json memory consumption as high as without chunksize #34548
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[mroeschke]

Thanks. Mind adding a unit test and whatsnew entry for version 1.2

[jreback]

yeah ideally this change should have broken a test (obviously) we don't have one. pls add a test which breaks on master and this change fixes (look in the original issues, where we want to replicate)

[robertwb]

I've added a test and addressed the comments.

[jreback]

can you also post the benchmarks for line read json (should show that this improves vs master)

[robertwb]

I don't expect it to improve raw performance, it's about memory usage for large files.

[jreback]

I don't expect it to improve raw performance, it's about memory usage for large files.

we have memory benchmarks to measure this

[jreback]

@robertwb if you can post the memory benchmarks

[simonjayhawkins]

the output is BENCHMARKS NOT SIGNIFICANTLY CHANGED. but IIUC io.json.ReadJSONLines.peakmem_read_json_lines_concat is reduced from 206M to 188M

maybe we need to change the benchmark to see a bigger effect.

(pandas-dev) simon@WIN-7LJRFMBC2ME:~/pandas/asv_bench$ asv continuous -f 1.1 master HEAD -b peakmem_read_json
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Building 4b856ee8 <json-chunksize> for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.........................................
·· Installing 4b856ee8 <json-chunksize> into conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
· Running 6 total benchmarks (2 commits * 1 environments * 3 benchmarks)
[  0.00%] · For pandas commit 4b856ee8 <json-chunksize> (round 1/1):
[  0.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 16.67%] ··· io.json.ReadJSONLines.peakmem_read_json_lines                                                           ok
[ 16.67%] ··· ========== ======
                index
              ---------- ------
                 int      250M
               datetime   250M
              ========== ======

[ 33.33%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_concat                                                    ok
[ 33.33%] ··· ========== ======
                index
              ---------- ------
                 int      188M
               datetime   188M
              ========== ======

[ 50.00%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_nrows                                                     ok
[ 50.00%] ··· ========== ======
                index
              ---------- ------
                 int      188M
               datetime   188M
              ========== ======

[ 50.00%] · For pandas commit 2733a109 <master> (round 1/1):
[ 50.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt......................................
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 66.67%] ··· io.json.ReadJSONLines.peakmem_read_json_lines                                                           ok
[ 66.67%] ··· ========== ======
                index
              ---------- ------
                 int      248M
               datetime   248M
              ========== ======

[ 83.33%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_concat                                                    ok
[ 83.33%] ··· ========== ======
                index
              ---------- ------
                 int      206M
               datetime   206M
              ========== ======

[100.00%] ··· io.json.ReadJSONLines.peakmem_read_json_lines_nrows                                                     ok
[100.00%] ··· ========== ======
                index
              ---------- ------
                 int      195M
               datetime   195M
              ========== ======


BENCHMARKS NOT SIGNIFICANTLY CHANGED.

[jreback]

the output is BENCHMARKS NOT SIGNIFICANTLY CHANGED. but IIUC io.json.ReadJSONLines.peakmem_read_json_lines_concat is reduced from 206M to 188M

maybe we need to change the benchmark to see a bigger effect.

so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.

[simonjayhawkins]

so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.

we do a concat of the chunks in the benchmark. maybe could just iterate through the chunks without the concat.

am currently running with chunksize=1 to compare.

[jreback]

so how do we know that this is working (and the previous was not), sure this is a difference, but wouldn't you expect a HUGE diffrence here? (e.g. maybe the benchmark should be reading with chunksize=1 or something) vs no chunksize.

we do a concat of the chunks in the benchmark. maybe could just iterate through the chunks without the concat.

am currently running with chunksize=1 to compare.

oh yeah the concat will increase memory :-> don't compare that

[simonjayhawkins]

changing the chunksize to 10 (1 fails) make a bit more difference (to the memory used on master!)

       before           after         ratio
     [2733a109]       [d731dc87]
     <master>         <json-chunksize>
-            224M             189M     0.84  io.json.ReadJSONLines.peakmem_read_json_lines_concat('int')
-            224M             188M     0.84  io.json.ReadJSONLines.peakmem_read_json_lines_concat('datetime')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.

[simonjayhawkins]

oh yeah the concat will increase memory :-> don't compare that

presumably we do the concat to compare the results with peakmem_read_json_lines

    def peakmem_read_json_lines(self, index):
        read_json(self.fname, orient="records", lines=True)

[jreback]

ok fair. @simonjayhawkins wouldn't object to changing the benchmarks (but this PR is fine).

[jreback]

thanks @robertwb

[jreback]

@meeseeksdev backport 1.2.x