[WIP] ENH: add Pyarrow csv engine

asv_bench/benchmarks/io/csv.py

@@ -146,7 +146,7 @@ def time_read_csv(self, bad_date_value):
 class ReadCSVSkipRows(BaseIO):
 
     fname = "__test__.csv"
-    params = ([None, 10000], ["c", "python"])
+    params = ([None, 10000], ["c", "pyarrow", "python"])
     param_names = ["skiprows", "engine"]
 
     def setup(self, skiprows, engine):
@@ -257,9 +257,18 @@ def time_read_csv_python_engine(self, sep, decimal, float_precision):
             names=list("abc"),
         )
 
+    def time_read_csv_pyarrow(self, sep, decimal, float_precision):
+        read_csv(
+            self.data(self.StringIO_input),
+            sep=sep,
+            header=None,
+            names=list("abc"),
+            engine="pyarrow",
+        )
+
 
 class ReadCSVEngine(StringIORewind):
-    params = ["c", "python"]
+    params = ["c", "python", "pyarrow"]
     param_names = ["engine"]
 
     def setup(self, engine):

doc/source/user_guide/io.rst

@@ -158,9 +158,11 @@ dtype : Type name or dict of column -> type, default ``None``
   (unsupported with ``engine='python'``). Use ``str`` or ``object`` together
   with suitable ``na_values`` settings to preserve and
   not interpret dtype.
-engine : {``'c'``, ``'python'``}
-  Parser engine to use. The C engine is faster while the Python engine is
-  currently more feature-complete.
+engine : {``'c'``, ``'pyarrow'``, ``'python'``}
+  Parser engine to use. The pyarrow engine is the most performant, followed by
+  the C engine, which in turn is faster than the python engine. However, the
+  pyarrow and C engine are currently less feature complete than their Python
+  counterpart.
 converters : dict, default ``None``
   Dict of functions for converting values in certain columns. Keys can either be
   integers or column labels.
@@ -1602,11 +1604,17 @@ Specifying ``iterator=True`` will also return the ``TextFileReader`` object:
 Specifying the parser engine
 ''''''''''''''''''''''''''''
 
-Under the hood pandas uses a fast and efficient parser implemented in C as well
-as a Python implementation which is currently more feature-complete. Where
-possible pandas uses the C parser (specified as ``engine='c'``), but may fall
-back to Python if C-unsupported options are specified. Currently, C-unsupported
-options include:
+Pandas currently supports three engines, the C engine, the python engine, and an optional
+pyarrow engine. The pyarrow engine is fastest, followed by the C and Python engines. However,
+the pyarrow engine is much less robust than the C engine, and the C engine is less feature-rich
+than the Python engine.
+
+Where possible pandas uses the C parser (specified as ``engine='c'``), but it may fall
+back to Python if C-unsupported options are specified. If pyarrow unsupported options are
+specified while using ``engine='pyarrow'``, the parser will throw an error.
+(a full list of unsupported options is available at ``pandas.io.parsers._pyarrow_unsupported``).
+
+Currently, C-unsupported options include:
 
 * ``sep`` other than a single character (e.g. regex separators)
 * ``skipfooter``

doc/source/whatsnew/v1.3.0.rst

@@ -13,6 +13,11 @@ including other versions of pandas.
 Enhancements
 ~~~~~~~~~~~~
 
+read_csv() now accepts pyarrow as an engine
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:func:`pandas.read_csv` now accepts engine="pyarrow" as an argument, allowing for faster csv parsing on multicore machines
+with pyarrow installed. See the :doc:`I/O docs </user_guide/io>` for more info. (:issue:`23697`)
 .. _whatsnew_130.read_csv_json_http_headers:
 
 Custom HTTP(s) headers when reading csv or json files