BUG, DOC: Allow custom line terminator with delim_whitespace=True

Title is self-explanatory.  Closes #12912.

Author: gfyoung <[email protected]>

Closes #12939 from gfyoung/delim-whitespace-fix and squashes the following commits:

78cf922 [gfyoung] MAINT: Refactor C engine tokenizing
62d6260 [gfyoung] BUG: Parse custom terminator with whitespace delimiter
fdbc768 [gfyoung] DOC: Add documentation for delim_whitespace

Author: gfyoung

doc/source/io.rst

@@ -97,6 +97,12 @@ sep : str, defaults to ``','`` for :func:`read_csv`, ``\t`` for :func:`read_tabl
   Regex example: ``'\\r\\t'``.
 delimiter : str, default ``None``
   Alternative argument name for sep.
+delim_whitespace : boolean, default False
+  Specifies whether or not whitespace (e.g. ``' '`` or ``'\t'``)
+  will be used as the delimiter. Equivalent to setting ``sep='\+s'``.
+  If this option is set to True, nothing should be passed in for the
+  ``delimiter`` parameter. This parameter is currently supported for
+  the C parser only.
 
 Column and Index Locations and Names
 ++++++++++++++++++++++++++++++++++++

doc/source/whatsnew/v0.18.1.txt

@@ -302,6 +302,7 @@ Bug Fixes
 - Bug in ``value_counts`` when ``normalize=True`` and ``dropna=True`` where nulls still contributed to the normalized count (:issue:`12558`)
 - Bug in ``Panel.fillna()`` ignoring ``inplace=True`` (:issue:`12633`)
 - Bug in ``read_csv`` when specifying ``names``, ```usecols``, and ``parse_dates`` simultaneously with the C engine (:issue:`9755`)
+- Bug in ``read_csv`` when specifying ``delim_whitespace=True`` and ``lineterminator`` simultaneously with the C engine (:issue:`12912`)
 - Bug in ``Series.rename``, ``DataFrame.rename`` and ``DataFrame.rename_axis`` not treating ``Series`` as mappings to relabel (:issue:`12623`).
 - Clean in ``.rolling.min`` and ``.rolling.max`` to enhance dtype handling (:issue:`12373`)
 - Bug in ``groupby`` where complex types are coerced to float (:issue:`12902`)

pandas/io/parsers.py

@@ -51,8 +51,13 @@
     file. For file URLs, a host is expected. For instance, a local file could
     be file ://localhost/path/to/table.csv
 %s
-delimiter : str, default None
+delimiter : str, default ``None``
     Alternative argument name for sep.
+delim_whitespace : boolean, default False
+    Specifies whether or not whitespace (e.g. ``' '`` or ``'\t'``) will be
+    used as the sep. Equivalent to setting ``sep='\+s'``. If this option
+    is set to True, nothing should be passed in for the ``delimiter``
+    parameter. This parameter is currently supported for the C parser only.
 header : int or list of ints, default 'infer'
     Row number(s) to use as the column names, and the start of the data.
     Default behavior is as if set to 0 if no ``names`` passed, otherwise

pandas/io/tests/test_parsers.py

@@ -3878,6 +3878,15 @@ def test_buffer_rd_bytes(self):
             except Exception as e:
                 pass
 
+    def test_delim_whitespace_custom_terminator(self):
+        # See gh-12912
+        data = """a b c~1 2 3~4 5 6~7 8 9"""
+        df = self.read_csv(StringIO(data), lineterminator='~',
+                           delim_whitespace=True)
+        expected = DataFrame([[1, 2, 3], [4, 5, 6], [7, 8, 9]],
+                             columns=['a', 'b', 'c'])
+        tm.assert_frame_equal(df, expected)
+
 
 class TestCParserHighMemory(CParserTests, CompressionTests, tm.TestCase):
     engine = 'c'