Skip to content

nrows incompatible with chunksize in read_csv #15755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
toobaz opened this issue Mar 21, 2017 · 0 comments
Closed

nrows incompatible with chunksize in read_csv #15755

toobaz opened this issue Mar 21, 2017 · 0 comments
Labels
Milestone

Comments

@toobaz
Copy link
Member

toobaz commented Mar 21, 2017

Code Sample, a copy-pastable example if possible

In [2]: pd.read_csv('/tmp/boh.csv', nrows=10, chunksize=2)
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-2-c652c48282ad> in <module>()
----> 1 pd.read_csv('/tmp/boh.csv', nrows=10, chunksize=2)

/home/nobackup/repo/pandas/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    651                     skip_blank_lines=skip_blank_lines)
    652 
--> 653         return _read(filepath_or_buffer, kwds)
    654 
    655     parser_f.__name__ = name

/home/nobackup/repo/pandas/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    391 
    392     if (nrows is not None) and (chunksize is not None):
--> 393         raise NotImplementedError("'nrows' and 'chunksize' cannot be used"
    394                                   " together yet.")
    395     elif nrows is not None:

NotImplementedError: 'nrows' and 'chunksize' cannot be used together yet.

(as introduced by #6774 )

Problem description

Supporting them together is not complicated - PR on its way.

@jreback jreback added this to the 0.20.0 milestone Mar 21, 2017
mattip pushed a commit to mattip/pandas that referenced this issue Apr 3, 2017
closes pandas-dev#15755

Author: Pietro Battiston <[email protected]>

Closes pandas-dev#15756 from toobaz/nrows_chunksize and squashes the following commits:

d0288e3 [Pietro Battiston] ENH: support "nrows" and "chunksize" together
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants