read_csv: chunksize clashes with nrows

[michaelaye]

If nrows are determined, the use of chunksize option does not create a TextFileReader object.

reader = pd.read_table(fname, sep=',',chunksize=4, na_values=['null'],
                       nrows=20)
type(reader)

pandas.core.frame.DataFrame

My suggestion:

• Either put into docs that they are not to be used concurrently
• or it's a feature request that they can.

I would find it useful to get chunks of x size, but only for the first n row of a huge file.

[jreback]

I think easiest ATM to simply raise NotImplementedError if both nrows and chunksize are specified.

Implemented is a bit non-trivial but would be useful I agree.

You want to do a PR for the NotImplementedError? Then we'll create an issue to implement this at some point.

[michaelaye]

I would like to but find io/parsers.py quite confusing.

read_csv is 'declared' here:

449 read_csv = _make_parser_function('read_csv', sep=',')
450 read_csv = Appender(_read_csv_doc)(read_csv)

_make_parser_function defines a parser_f on line 311, and that definition has both the chunksize and nrows option, but is it true, that they are in no case, so for none of the generated parsers, work together? In that case should I implemented right there, at parser_f definition?

[jreback]

look here: https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L230

[myidealab]

Is there a workaround for this issue? I am trying to pass in different parameters for testing and production.

Testing: chunksize=None, nrows=n
Production: chunksize=i, nrows =None

Production works fine, but when I try to implement the testing version, I receive the same error as others: NotImplementedError: 'nrows' and 'chunksize' cannot be used together yet.

**Edit: I ended up utilizing a conditional statement and added a parameter for version type.