Skip to content

BUG: skip_blank_lines ignored by read_fwf #37803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -517,6 +517,7 @@ I/O
- Bug in :meth:`DataFrame.to_hdf` was not dropping missing rows with ``dropna=True`` (:issue:`35719`)
- Bug in :func:`read_html` was raising a ``TypeError`` when supplying a ``pathlib.Path`` argument to the ``io`` parameter (:issue:`37705`)
- :meth:`to_excel` and :meth:`to_markdown` support writing to fsspec URLs such as S3 and Google Cloud Storage (:issue:`33987`)
- Bug in :meth:`read_fw` was not skipping blank lines (even with ``skip_blank_lines=True``) (:issue:`37758`)

Plotting
^^^^^^^^
Expand Down
13 changes: 13 additions & 0 deletions pandas/io/parsers.py
Original file line number Diff line number Diff line change
Expand Up @@ -3750,6 +3750,19 @@ def _make_reader(self, f):
self.infer_nrows,
)

def _remove_empty_lines(self, lines) -> List:
"""
Returns the list of lines without the empty ones. With fixed-width
fields, empty lines become arrays of empty strings.

See PythonParser._remove_empty_lines.
"""
return [
line
for line in lines
if any(not isinstance(e, str) or e.strip() for e in line)
]


def _refine_defaults_read(
dialect: Union[str, csv.Dialect],
Expand Down
45 changes: 45 additions & 0 deletions pandas/tests/io/parser/test_read_fwf.py
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,51 @@ def test_fwf_comment(comment):
tm.assert_almost_equal(result, expected)


def test_fwf_skip_blank_lines():
data = """

A B C D

201158 360.242940 149.910199 11950.7
201159 444.953632 166.985655 11788.4


201162 502.953953 173.237159 12468.3

"""
result = read_fwf(StringIO(data), skip_blank_lines=True)
expected = DataFrame(
[
[201158, 360.242940, 149.910199, 11950.7],
[201159, 444.953632, 166.985655, 11788.4],
[201162, 502.953953, 173.237159, 12468.3],
],
columns=["A", "B", "C", "D"],
)
tm.assert_frame_equal(result, expected)

data = """\
A B C D
201158 360.242940 149.910199 11950.7
201159 444.953632 166.985655 11788.4


201162 502.953953 173.237159 12468.3
"""
result = read_fwf(StringIO(data), skip_blank_lines=False)
expected = DataFrame(
[
[201158, 360.242940, 149.910199, 11950.7],
[201159, 444.953632, 166.985655, 11788.4],
[np.nan, np.nan, np.nan, np.nan],
[np.nan, np.nan, np.nan, np.nan],
[201162, 502.953953, 173.237159, 12468.3],
],
columns=["A", "B", "C", "D"],
)
tm.assert_frame_equal(result, expected)


@pytest.mark.parametrize("thousands", [",", "#", "~"])
def test_fwf_thousands(thousands):
data = """\
Expand Down