Skip to content

Commit 9506a3c

Browse files
committed
PERF: don't create the skiprows set if using the c-parser
closes #13005
1 parent 0843911 commit 9506a3c

File tree

2 files changed

+7
-4
lines changed

2 files changed

+7
-4
lines changed

doc/source/whatsnew/v0.18.1.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -408,7 +408,7 @@ Performance Improvements
408408

409409
- Improved speed of SAS reader (:issue:`12656`)
410410
- Performance improvements in ``.groupby(..).cumcount()`` (:issue:`11039`)
411-
411+
- Improved memory usage in ``pd.read_csv()`` when using ``skiprows=an_integer`` (:issue:`13005`)
412412

413413
- Improved performance of ``DataFrame.to_sql`` when checking case sensitivity for tables. Now only checks if table has been created correctly when table name is not lower case. (:issue:`12876`)
414414
- Improved performance of ``Period`` construction and plotting of ``Period``s. (:issue:`12903`, :issue:`11831`)

pandas/io/parsers.py

+6-3
Original file line numberDiff line numberDiff line change
@@ -775,9 +775,12 @@ def _clean_options(self, options, engine):
775775
# Converting values to NA
776776
na_values, na_fvalues = _clean_na_values(na_values, keep_default_na)
777777

778-
if com.is_integer(skiprows):
779-
skiprows = lrange(skiprows)
780-
skiprows = set() if skiprows is None else set(skiprows)
778+
# handle skiprows; this is internally handled by the
779+
# c-engine, so only need for python parsers
780+
if engine != 'c':
781+
if com.is_integer(skiprows):
782+
skiprows = lrange(skiprows)
783+
skiprows = set() if skiprows is None else set(skiprows)
781784

782785
# put stuff back
783786
result['names'] = names

0 commit comments

Comments
 (0)