Refactor read_csv bug fix with PR comments (#18186)

Author: sam-cohan

doc/source/whatsnew/v0.21.1.txt

@@ -56,15 +56,6 @@ Documentation Changes
 
 Bug Fixes
 ~~~~~~~~~
-- Bug in ``DataFrame.resample(...).apply(...)`` when there is a callable that returns different columns (:issue:`15169`)
-- Bug in :class:`TimedeltaIndex` subtraction could incorrectly overflow when ``NaT`` is present (:issue:`17791`)
-- Bug in :class:`DatetimeIndex` subtracting datetimelike from DatetimeIndex could fail to overflow (:issue:`18020`)
-- Bug in ``pd.Series.rolling.skew()`` and ``rolling.kurt()`` with all equal values has floating issue (:issue:`18044`)
-- Bug in ``pd.DataFrameGroupBy.count()`` when counting over a datetimelike column (:issue:`13393`)
-- Bug in ``pd.concat`` when empty and non-empty DataFrames or Series are concatenated (:issue:`18178` :issue:`18187`)
-- Bug in :class:`IntervalIndex` constructor when a list of intervals is passed with non-default ``closed`` (:issue:`18334`)
-- Bug in :meth:`IntervalIndex.copy` when copying and ``IntervalIndex`` with non-default ``closed`` (:issue:`18339`)
-- Bug in ``pd.read_csv`` when reading numeric category fields with high cardinality (:issue `18186`)
 
 Conversion
 ^^^^^^^^^^
@@ -91,6 +82,7 @@ I/O
 - Bug in class:`~pandas.io.stata.StataReader` not converting date/time columns with display formatting addressed (:issue:`17990`). Previously columns with display formatting were normally left as ordinal numbers and not converted to datetime objects.
 - Bug in :func:`read_csv` when reading a compressed UTF-16 encoded file (:issue:`18071`)
 - Bug in :func:`read_csv` for handling null values in index columns when specifying ``na_filter=False`` (:issue:`5239`)
+- Bug in :func:`read_csv` when reading numeric category fields with high cardinality (:issue:`18186`)
 - Bug in :meth:`DataFrame.to_csv` when the table had ``MultiIndex`` columns, and a list of strings was passed in for ``header`` (:issue:`5539`)
 - :func:`read_parquet` now allows to specify the columns to read from a parquet file (:issue:`18154`)
 - :func:`read_parquet` now allows to specify kwargs which are passed to the respective engine (:issue:`18216`)

pandas/_libs/parsers.pyx

@@ -2227,7 +2227,7 @@ def _concatenate_chunks(list chunks):
     for name in names:
         arrs = [chunk.pop(name) for chunk in chunks]
         # Check each arr for consistent types.
-        dtypes = set([a.dtype for a in arrs])
+        dtypes = {a.dtype for a in arrs}
         numpy_dtypes = {x for x in dtypes if not is_categorical_dtype(x)}
         if len(numpy_dtypes) > 1:
             common_type = np.find_common_type(numpy_dtypes, [])

pandas/tests/io/parser/dtypes.py

@@ -114,15 +114,14 @@ def test_categorical_dtype(self):
         actual = self.read_csv(StringIO(data), dtype='category')
         tm.assert_frame_equal(actual, expected)
 
-    @pytest.mark.slow
     def test_categorical_dtype_high_cardinality_numeric(self):
         # GH 18186
-        data = sorted([str(i) for i in range(10**6)])
-        expected = pd.DataFrame({'a': Categorical(data, ordered=True)})
+        data = np.sort([str(i) for i in range(524289)])
+        expected = DataFrame({'a': Categorical(data, ordered=True)})
         actual = self.read_csv(StringIO('a\n' + '\n'.join(data)),
                                dtype='category')
-        actual.a.cat.reorder_categories(sorted(actual.a.cat.categories),
-                                        ordered=True, inplace=True)
+        actual["a"] = actual["a"].cat.reorder_categories(
+            np.sort(actual.a.cat.categories), ordered=True)
         tm.assert_frame_equal(actual, expected)
 
     def test_categorical_dtype_encoding(self):