DOC: low_memory in read_csv #13293
DOC: low_memory in read_csv #13293
Conversation
| low_memory : boolean, default ``True`` | ||
| Internally process the file in chunks, resulting in lower memory use | ||
| while parsing, but possibly mixed type inference. To ensure no mixed | ||
| types either set False, or specify the type with the `dtype` parameter. |
There was a problem hiding this comment.
double back-ticks on False
Current coverage is 84.20%@@ master #13293 diff @@
==========================================
Files 138 138
Lines 50587 50587
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
Hits 42593 42593
Misses 7994 7994
Partials 0 0
|
|
@jreback - updated |
|
thanks! |
| @@ -169,6 +169,13 @@ skipfooter : int, default ``0`` | |||
| Number of lines at bottom of file to skip (unsupported with engine='c'). | |||
| nrows : int, default ``None`` | |||
| Number of rows of file to read. Useful for reading pieces of large files. | |||
| low_memory : boolean, default ``True`` | |||
| Internally process the file in chunks, resulting in lower memory use | |||
| while parsing, but possibly mixed type inference. To ensure no mixed | |||
There was a problem hiding this comment.
What does 'resulting in [...] mixed type inference' mean? Is it implementation-specific which type is chosen? Is it deterministic?
There was a problem hiding this comment.
It is deterministic - types are consistently inferred based on what's in the data. That said, the internal chunksize is not a fixed number of rows, but instead bytes, so whether you can a mixed dtype warning or not can feel a bit random.
from io import StringIO
# just ints - no warning
pd.read_csv(StringIO(
'\n'.join(str(x) for x in range(10000)))
).dtypes
Out[24]:
0 int64
dtype: object
# mixed dtype - but fits into a single chunk, so no warning and all parsed as strings
pd.read_csv(StringIO(
'\n'.join([str(x) for x in range(10000)] + ['a string']))
).dtypes
Out[26]:
0 object
dtype: object
# mixed dtype - doesn't fit into a single chunk
pd.read_csv(StringIO(
'\n'.join([str(x) for x in range(1000000)] + ['a string']))
).dtypes
DtypeWarning: Columns (0) have mixed types.
Specify dtype option on import or set low_memory=False.
Out[27]:
0 object
dtype: objectThere was a problem hiding this comment.
I see. Thank you for the explanation and the example!
There was a problem hiding this comment.
To complete the example, I see:
df=pd.read_csv(StringIO('\n'.join([str(x) for x in range(1000000)] + ['a string'])))
DtypeWarning: Columns (0) have mixed types. Specify dtype option on import or set low_memory=False.
type(df.loc[524287,'0'])
Out[50]: int
type(df.loc[524288,'0'])
Out[51]: strThe first part of the csv data was seen as only int, so converted to int,
the second part also had a string, so all entries were kept as string.
git diff upstream/master | flake8 --diff