BUG: read_excel skips single-column empty rows

[ahawryluk]

Ref #39808

• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

This is the fix proposed in the discussion of #40095

[rhshadrach]

Changes look good - two requests below.

[gfyoung]

Why not expose this parameter in read_excel ?

[ahawryluk]

We tried that (#40095) but the behaviour when skip_blank_lines=True isn't very useful---it only skips empty spreadsheet rows if the entire spreadsheet is contained in column A. The original intent of skip_blank_lines was to skip \n lines in CSV files, but not lines with empty data elements such as ""\n or ,,,,\n, so there isn't an equivalent feature for spreadsheets.

[gfyoung]

I see. Can we make a comment about this and reference your original PR?

[ahawryluk]

For sure. Do you mean an inline comment in the code, or in the description of this pull request?

[lithomas1]

Not sure if this closes the original issue, as the parameter would have to be exposed in order to truly fix.
I think modifying this function

pandas/pandas/io/parsers/python_parser.py

Line 596 in fad96e1

def _is_line_empty(self, line):

might be able to fix the issue, but it could also introduce some side effects for CSV parsing.

[ahawryluk]

@lithomas1 I'm nervous about breaking the CSV behaviour, because it seems deliberate. I found one test where an input line ,, is supposed to produce a row of NaNs:

pandas/pandas/tests/io/parser/common/test_common_basic.py

Lines 412 to 414 in 154026c

	",,",
	{"names": ["Dummy", "X", "Dummy_2"], "usecols": ["X"]},
	DataFrame(columns=["X"], index=[0], dtype=np.float64),

Maybe this behaviour should be documented better?

[rhshadrach]

lgtm; @lithomas1 - agreed that this PR does not fully close the issue as stated there, however I'm -0 on adding the skip_blank_lines argument to read_excel. read_csv needs this because you can't tell a blank line from e.g. ,,,, from the resulting DataFrame; this does not exist for excel files and it is straightforward to find and drop blank lines with df.dropna(how='all').

@ahawryluk - can you merge master

[rhshadrach]

@simonjayhawkins @jreback - This is a bit of an odd case, wanted to get some eyes to make sure it's being handled appropriately. This PR fixes the following bug (not a regression, AFAICT) on master: when reading an excel file, blank rows would be skipped if and only if a single column is being read. This PR makes it so that blank rows are never skipped.

However, the previous workaround was to supply skip_blank_lines=False in **kwds; this argument was removed in 1.1 and so is no longer available. I'm -0 on adding skip_blank_lines back in (see #40214 (review)).

Since this is a rather localized case, my thinking is to have this be a bugfix for 1.3 and leave the regression in 1.1.x and 1.2.x unresolved.

[simonjayhawkins]

Since this is a rather localized case, my thinking is to have this be a bugfix for 1.3 and leave the regression in 1.1.x and 1.2.x unresolved.

sgtm. does the OP need to change to not close the issue?

[ahawryluk]

@simonjayhawkins, thanks for taking a look at this. My preference is to close #39808 based on

• the OP of Add back skip_blank_lines to read_excel in pandas v>1.1.4 #39808 (comment) prefers the current solution now that we understand how skip_blank_lines really works
• IIUC, @rhshadrach and I don't think skip_blank_lines should be added to read_excel in the future

But I'm happy to edit the PR if everyone prefers to leave #39808 open.

[simonjayhawkins]

• IIUC, @rhshadrach and I don't think skip_blank_lines should be added to read_excel in the future

The issue could be closed once that is agreed as they way forward on the issue itself.

[jreback]

lgtm. back to you @rhshadrach

[jreback]

for 1.3

[rhshadrach]

Thanks @ahawryluk! I've left the issue open for now, but will put my thoughts there shortly.