Skip to content

BUG: output formatting with to_html(), index=False and/or index_names=False (#22579, #22747) #22655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 109 commits into from
Jan 1, 2019
Merged

BUG: output formatting with to_html(), index=False and/or index_names=False (#22579, #22747) #22655

Show file tree
Hide file tree
Changes from 100 commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
6adc266
BUG: header alignment
simonjayhawkins Sep 10, 2018
a30da56
add test_to_html_index_name_single_index
simonjayhawkins Sep 10, 2018
b444fa2
add placeholders for multiIndex tests
simonjayhawkins Sep 10, 2018
c61ea4a
add regression test
simonjayhawkins Sep 10, 2018
863b6d6
add regression test
simonjayhawkins Sep 10, 2018
b44d4ff
add regression test
simonjayhawkins Sep 10, 2018
d5c37e3
add regression test
simonjayhawkins Sep 10, 2018
6b441df
prep test_to_html_index_name_multi_index_both_index_false
simonjayhawkins Sep 10, 2018
dcf74a5
add regression test
simonjayhawkins Sep 10, 2018
dd07605
add regression test
simonjayhawkins Sep 10, 2018
4605a4e
add regression test
simonjayhawkins Sep 10, 2018
dd825f3
add regression test
simonjayhawkins Sep 10, 2018
b7fe95c
add regression test
simonjayhawkins Sep 10, 2018
dec609d
add regression test
simonjayhawkins Sep 10, 2018
5b9bc6e
add regression test
simonjayhawkins Sep 10, 2018
a725108
move regression test
simonjayhawkins Sep 10, 2018
d7e8237
remove duplicated regression tests
simonjayhawkins Sep 10, 2018
47fd132
add regression test
simonjayhawkins Sep 10, 2018
4e48a32
add regression test
simonjayhawkins Sep 10, 2018
30ac94e
add failing test
simonjayhawkins Sep 10, 2018
5f6a8d1
split and rename tests
simonjayhawkins Sep 10, 2018
1b4c5dc
fix failing test
simonjayhawkins Sep 10, 2018
82d57eb
add test (failing)
simonjayhawkins Sep 10, 2018
cfa7570
fix failing test
simonjayhawkins Sep 10, 2018
9f884fb
refactor
simonjayhawkins Sep 10, 2018
25e8103
add regression test (failing)
simonjayhawkins Sep 10, 2018
d7ec7a3
fix failing test
simonjayhawkins Sep 11, 2018
17f66ac
add test
simonjayhawkins Sep 11, 2018
92d0c9a
add test (failing)
simonjayhawkins Sep 11, 2018
e6da59b
revert html.py to upstream/master
simonjayhawkins Sep 11, 2018
96b181e
re-apply minimal code changes
simonjayhawkins Sep 11, 2018
e737cd6
minimise changes further
simonjayhawkins Sep 11, 2018
9dbb6ed
refactor
simonjayhawkins Sep 11, 2018
130873f
refactor
simonjayhawkins Sep 11, 2018
fc8851d
fix failing test
simonjayhawkins Sep 11, 2018
2ace532
refactor
simonjayhawkins Sep 11, 2018
49177d7
fix pep8
simonjayhawkins Sep 11, 2018
5cbb8c5
refactor
simonjayhawkins Sep 11, 2018
1db45f2
WIP parametrize tests
simonjayhawkins Sep 11, 2018
cf883d9
WIP parametrize tests
simonjayhawkins Sep 11, 2018
3842c52
fix typo
simonjayhawkins Sep 11, 2018
8722603
refactor test
simonjayhawkins Sep 11, 2018
d7c2e20
WIP parametrize tests
simonjayhawkins Sep 11, 2018
2aed4ec
WIP parametrize tests
simonjayhawkins Sep 11, 2018
f8191f0
WIP parametrize tests
simonjayhawkins Sep 12, 2018
c21f7cd
add test (failing)
simonjayhawkins Sep 12, 2018
3263cc1
fix failing test
simonjayhawkins Sep 13, 2018
528fcbc
add test
simonjayhawkins Sep 13, 2018
fae4070
refactor
simonjayhawkins Sep 13, 2018
c3ab8d2
refactor test
simonjayhawkins Sep 13, 2018
0d9330e
replace single with standard
simonjayhawkins Sep 13, 2018
ebf711b
refactor tests
simonjayhawkins Sep 13, 2018
b798429
refactor test
simonjayhawkins Sep 13, 2018
23e3204
refactor test
simonjayhawkins Sep 13, 2018
9fd31b3
add tests (failing)
simonjayhawkins Sep 13, 2018
22396e2
fix failing tests
simonjayhawkins Sep 13, 2018
aee9b7e
minimise changes
simonjayhawkins Sep 13, 2018
aae1632
WIP add test fixtures
simonjayhawkins Sep 13, 2018
3989085
WIP add test fixtures
simonjayhawkins Sep 14, 2018
21a70ed
WIP add test fixtures
simonjayhawkins Sep 14, 2018
85490a9
WIP add test fixtures
simonjayhawkins Sep 14, 2018
2452aff
WIP add test fixtures
simonjayhawkins Sep 14, 2018
9a0512c
WIP add test fixtures
simonjayhawkins Sep 14, 2018
f2b5f75
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Sep 15, 2018
f94e336
add test
simonjayhawkins Sep 15, 2018
db32821
refactor test
simonjayhawkins Sep 15, 2018
2546d82
move read_file fixture factory into conftest.py
simonjayhawkins Sep 15, 2018
29867d4
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Sep 15, 2018
483ace8
update expected output
simonjayhawkins Sep 17, 2018
96b5442
fix failing tests
simonjayhawkins Sep 17, 2018
00d452c
refactor
simonjayhawkins Sep 17, 2018
3d1d5c4
refactor
simonjayhawkins Sep 18, 2018
5dcd146
requested change
simonjayhawkins Sep 19, 2018
5327098
add comments
simonjayhawkins Sep 19, 2018
0961db7
refactor
simonjayhawkins Sep 20, 2018
9b8e203
(wip) resolve merge conflicts
simonjayhawkins Nov 16, 2018
677472a
(wip) remove conftest.py
simonjayhawkins Nov 16, 2018
5ec0c57
(wip) refactor test
simonjayhawkins Nov 16, 2018
72d485d
(wip) refactor test
simonjayhawkins Nov 16, 2018
dd84b8d
(wip) add tests with truncation
simonjayhawkins Nov 17, 2018
57c035f
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Nov 18, 2018
9b7eaae
(wip) parametrize truncation tests
simonjayhawkins Nov 18, 2018
7383c8a
(wip) refactor test helper
simonjayhawkins Nov 18, 2018
d89a180
(wip) removed unused header parametrization
simonjayhawkins Nov 18, 2018
24ba91c
(wip) refactor
simonjayhawkins Nov 18, 2018
a4feea8
add whatsnew entry
simonjayhawkins Nov 18, 2018
b4a2b10
resolve merge conflicts
simonjayhawkins Nov 18, 2018
e00cbaf
requested changes
simonjayhawkins Nov 19, 2018
8d8b138
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Nov 20, 2018
767541b
changes requested
simonjayhawkins Nov 26, 2018
b8f1e6a
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Nov 26, 2018
9eb162f
requested changes
simonjayhawkins Nov 27, 2018
48d118b
resolve merge conflicts
simonjayhawkins Nov 27, 2018
28fab9d
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 3, 2018
d60fa0b
double backticks on MultiIndex
simonjayhawkins Dec 3, 2018
d3986f0
add code comments
simonjayhawkins Dec 9, 2018
1febf76
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 9, 2018
6d60064
additional test case for gh-22783
simonjayhawkins Dec 10, 2018
1f61968
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 10, 2018
bd815e7
resolve merge conflicts
simonjayhawkins Dec 13, 2018
7da52a1
add test case
simonjayhawkins Dec 14, 2018
0a0f82f
make show_col_idx_names class property
simonjayhawkins Dec 14, 2018
bc7f8c7
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 14, 2018
8d2d68a
parametrize truncation tests
simonjayhawkins Dec 14, 2018
d2e233e
fix whitespace
simonjayhawkins Dec 14, 2018
bdaa279
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 14, 2018
b7e4f54
make row_levels class property
simonjayhawkins Dec 14, 2018
613ce00
Merge remote-tracking branch 'upstream/master' into issue22579
simonjayhawkins Dec 28, 2018
5b635e4
move data files
simonjayhawkins Dec 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v0.24.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1497,6 +1497,8 @@ Notice how we now instead output ``np.nan`` itself instead of a stringified form
- :func:`read_sas()` will correctly parse sas7bdat files with data page types having also bit 7 set (so page type is 128 + 256 = 384) (:issue:`16615`)
- Bug in :meth:`detect_client_encoding` where potential ``IOError`` goes unhandled when importing in a mod_wsgi process due to restricted access to stdout. (:issue:`21552`)
- Bug in :func:`to_html()` with ``index=False`` misses truncation indicators (...) on truncated DataFrame (:issue:`15019`, :issue:`22783`)
- Bug in :func:`to_html()` with ``index=False`` when both columns and row index are ``MultiIndex`` (:issue:`22579`)
- Bug in :func:`to_html()` with ``index_names=False`` displaying index name (:issue:`22747`)
- Bug in :func:`DataFrame.to_string()` that broke column alignment when ``index=False`` and width of first column's values is greater than the width of first column's header (:issue:`16839`, :issue:`13032`)
- Bug in :func:`DataFrame.to_string()` that caused representations of :class:`DataFrame` to not take up the whole window (:issue:`22984`)
- Bug in :func:`DataFrame.to_csv` where a single level MultiIndex incorrectly wrote a tuple. Now just the value of the index is written (:issue:`19589`).
Expand Down
91 changes: 76 additions & 15 deletions pandas/io/formats/html.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ def __init__(self, formatter, classes=None, notebook=False, border=None,
self.border = border
self.table_id = table_id
self.render_links = render_links
# see gh-22579
# Column misalignment also occurs for
# a standard index when the columns index is named.
# Determine if ANY column names need to be displayed
# since if the row index is not displayed a column of
# blank cells need to be included before the DataFrame values.
self.show_col_idx_names = all((self.fmt.has_column_names,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for questions but still trying to wrap my head around implementation. Based off of the comment, why is this all here and not any? Wouldn't any of these require there to be a cell where a column index name would be placed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

index=False with a single level row index and multi-level columns index with named columns but not all named...

index = pd.MultiIndex.from_product([['a','b'], ['c','d'], ['e','f']], names=[
                 'foo',None, 'baz'])
df = pd.DataFrame(np.arange(64).reshape(8,8), columns=index)
result = df.to_html(max_rows=4, max_cols=4, index=False)
print(result)
foo a ... b
c ... d
baz e f ... e f
0 1 6 7
8 9 14 15
48 49 54 55
56 57 62 63

Note: missing truncation indicators in data now fixed in master.

the misalignment of the column names is due to the logic being applied within the level generating loop..

pandas/pandas/io/formats/html.py

Lines 270 to 275 in d43ac97

name = self.columns.names[lnum]
row = [''] * (row_levels - 1) + ['' if name is None else
pprint_thing(name)]
if row == [""] and self.fmt.index is False:
row = []

hence class-level variable needed to check if ANY names need to be displayed to determine alignment.

ALL condition is to determine in ANY names should be displayed given the to_html parameters and uses similar logic as to_string etc.

pandas/pandas/io/formats/format.py

Lines 796 to 803 in d43ac97

def _get_formatted_index(self, frame):
# Note: this is only used by to_string() and to_latex(), not by
# to_html().
index = frame.index
columns = frame.columns
show_index_names = self.show_index_names and self.has_index_names
show_col_names = (self.show_index_names and self.has_column_names)

and the rows in to_html..

pandas/pandas/io/formats/html.py

Lines 307 to 309 in d43ac97

if all((self.fmt.has_index_names,
self.fmt.index,
self.fmt.show_index_names)):

There is currently no test to explicitly cover this example. so i think the best way forward is to fully parameterize the truncation tests in line with the parametrized basic_alignment tests for added assurance.

i'll make show_col_idx_names a class property for clarity and add a note to refactor and 'inherit' from DataFrameFormatter class. inherit quoted since HTMLFormatter class is not directly inherited from DataFrameFormatter. in the first refactor just use mock inheritence like..

pandas/pandas/io/formats/html.py

Lines 46 to 48 in d43ac97

@property
def is_truncated(self):
return self.fmt.is_truncated

self.fmt.show_index_names,
self.fmt.header))

@property
def is_truncated(self):
Expand Down Expand Up @@ -201,7 +210,22 @@ def write_result(self, buf):

def _write_header(self, indent):
truncate_h = self.fmt.truncate_h
row_levels = self.frame.index.nlevels
# see gh-22579
# Column Offset Bug with to_html(index=False) with
# MultiIndex Columns and Index.
# Column misalignment also occurs for
# a standard index when the columns index is named.
# If the row index is not displayed a column of
# blank cells need to be included before the DataFrame values.
# However, in this code block only the placement of the truncation
# indicators within the header is affected by this.
# TODO: refactor to class property as row_levels also used in
# _write_regular_rows and _write_hierarchical_rows
if self.fmt.index:
row_levels = self.frame.index.nlevels
else:
row_levels = 1 if self.show_col_idx_names else 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What gets shown when this evaluates to True?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

row_levels is used to determine the number of header tags, number of blank cells before the column header etc. it is the column number(of the html table) where the data starts.

if showing the (row) index (if self.fmt.index) the number of cells in the html table before the data is the number of levels in the row index (row_levels = self.frame.index.nlevels).

if not showing the (row) index, the number of cells in the html table before the data is 1 (blank) if the column names are displayed and none if the column names are not displayed or the column index is not named (self.show_col_idx_names)

self.show_col_idx_names = all((self.fmt.has_column_names, self.fmt.show_index_names, self.fmt.header))

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd

What gets shown when this evaluates to True?

row_levels=1 in _write_header only affects the placement of the truncation indicators for a single level named columns index.

Changing the 1 to a 0 does not fail any tests so i've added an additional test to gh-22783 to ensure we get a test failure if row-levels is not 1.


if not self.fmt.header:
# write nothing
return indent
Expand Down Expand Up @@ -267,12 +291,26 @@ def _write_header(self, indent):
values = (values[:ins_col] + [u('...')] +
values[ins_col:])

name = self.columns.names[lnum]
row = [''] * (row_levels - 1) + ['' if name is None else
pprint_thing(name)]

if row == [""] and self.fmt.index is False:
row = []
# see gh-22579
# Column Offset Bug with to_html(index=False) with
# MultiIndex Columns and Index.
# Initially fill row with blank cells before column names.
# TODO: Refactor to remove code duplication with code
# block below for standard columns index.
row = [''] * (row_levels - 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be worth it to make this a function / method as it appears to repeat below.

can you add a comment on what this is doing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fixes for both the issues with the standard index and and issues with the multiindex have been purposefully made to use similar code to make the subsequent refactor simpler.

imo the refactor should create a _get_formatted_column_labels method for code parity with the DataFrameFormatter class but that refactor is outside the scope of this PR. the DataFrameFormatter class also has a _get_column_name_list which could be used.

However, i've not yet done this refactor because I want to get the bugs fixed first and the parametric tests in place for added assurance. I really wanted to limit the scope of this PR to the bug fixes since there is also a couple of issues with the header=False and sparsify=False which really need sorting before refactoring.

I will happily add a function but it would only be temporary. please let me know your thoughts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok that’s fine for then
pls add comments though and a TODO here as well

if self.fmt.index or self.show_col_idx_names:
# see gh-22747
# If to_html(index_names=False) do not show columns
# index names.
# TODO: Refactor to use _get_column_name_list from
# DataFrameFormatter class and create a
# _get_formatted_column_labels function for code
# parity with DataFrameFormatter class.
if self.fmt.show_index_names:
name = self.columns.names[lnum]
row.append(pprint_thing(name or ''))
else:
row.append('')

tags = {}
j = len(row)
Expand All @@ -287,17 +325,27 @@ def _write_header(self, indent):
self.write_tr(row, indent, self.indent_delta, tags=tags,
header=True)
else:
if self.fmt.index:
row = [''] * (self.frame.index.nlevels - 1)
row.append(self.columns.name or '')
else:
row = []
# see gh-22579
# Column misalignment also occurs for
# a standard index when the columns index is named.
# Initially fill row with blank cells before column names.
# TODO: Refactor to remove code duplication with code block
# above for columns MultiIndex.
row = [''] * (row_levels - 1)
if self.fmt.index or self.show_col_idx_names:
# see gh-22747
# If to_html(index_names=False) do not show columns
# index names.
# TODO: Refactor to use _get_column_name_list from
# DataFrameFormatter class.
if self.fmt.show_index_names:
row.append(self.columns.name or '')
else:
row.append('')
row.extend(self.columns)
align = self.fmt.justify

if truncate_h:
if not self.fmt.index:
row_levels = 0
ins_col = row_levels + self.fmt.tr_col_num
row.insert(ins_col, '...')

Expand Down Expand Up @@ -348,7 +396,14 @@ def _write_regular_rows(self, fmt_values, indent):
index_values = self.fmt.tr_frame.index.format()
row_levels = 1
else:
row_levels = 0
# see gh-22579
# Column misalignment also occurs for
# a standard index when the columns index is named.
# row_levels is used for the number of <th> cells and
# the placement of the truncation indicators.
# TODO: refactor to class property as row_levels also used in
# _write_header and _write_hierarchical_rows
row_levels = 1 if self.show_col_idx_names else 0

row = []
for i in range(nrows):
Expand All @@ -361,6 +416,12 @@ def _write_regular_rows(self, fmt_values, indent):
row = []
if self.fmt.index:
row.append(index_values[i])
# see gh-22579
# Column misalignment also occurs for
# a standard index when the columns index is named.
# Add blank cell before data cells.
elif self.show_col_idx_names:
row.append('')
row.extend(fmt_values[j][i] for j in range(self.ncols))

if truncate_h:
Expand Down
76 changes: 76 additions & 0 deletions pandas/tests/io/formats/data/gh22579_expected_output.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<table border="1" class="dataframe">
<thead>
<tr>
<th colspan="2" halign="left">a</th>
<th colspan="2" halign="left">b</th>
</tr>
<tr>
<th>c</th>
<th>d</th>
<th>c</th>
<th>d</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>10</td>
<td>10</td>
<td>10</td>
</tr>
<tr>
<td>1</td>
<td>11</td>
<td>11</td>
<td>11</td>
</tr>
<tr>
<td>2</td>
<td>12</td>
<td>12</td>
<td>12</td>
</tr>
<tr>
<td>3</td>
<td>13</td>
<td>13</td>
<td>13</td>
</tr>
<tr>
<td>4</td>
<td>14</td>
<td>14</td>
<td>14</td>
</tr>
<tr>
<td>5</td>
<td>15</td>
<td>15</td>
<td>15</td>
</tr>
<tr>
<td>6</td>
<td>16</td>
<td>16</td>
<td>16</td>
</tr>
<tr>
<td>7</td>
<td>17</td>
<td>17</td>
<td>17</td>
</tr>
<tr>
<td>8</td>
<td>18</td>
<td>18</td>
<td>18</td>
</tr>
<tr>
<td>9</td>
<td>19</td>
<td>19</td>
<td>19</td>
</tr>
</tbody>
</table>
66 changes: 66 additions & 0 deletions pandas/tests/io/formats/data/gh22579_truncation_index_false_named.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<table border="1" class="dataframe">
<thead>
<tr>
<th>foo</th>
<th colspan="2" halign="left">a</th>
<th>...</th>
<th colspan="2" halign="left">b</th>
</tr>
<tr>
<th></th>
<th colspan="2" halign="left">c</th>
<th>...</th>
<th colspan="2" halign="left">d</th>
</tr>
<tr>
<th>baz</th>
<th>e</th>
<th>f</th>
<th>...</th>
<th>e</th>
<th>f</th>
</tr>
</thead>
<tbody>
<tr>
<th></th>
<td>0</td>
<td>1</td>
<td>...</td>
<td>6</td>
<td>7</td>
</tr>
<tr>
<th></th>
<td>8</td>
<td>9</td>
<td>...</td>
<td>14</td>
<td>15</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th></th>
<td>48</td>
<td>49</td>
<td>...</td>
<td>54</td>
<td>55</td>
</tr>
<tr>
<th></th>
<td>56</td>
<td>57</td>
<td>...</td>
<td>62</td>
<td>63</td>
</tr>
</tbody>
</table>
58 changes: 58 additions & 0 deletions pandas/tests/io/formats/data/gh22579_truncation_index_false_unnamed.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
<table border="1" class="dataframe">
<thead>
<tr>
<th colspan="2" halign="left">a</th>
<th>...</th>
<th colspan="2" halign="left">b</th>
</tr>
<tr>
<th colspan="2" halign="left">c</th>
<th>...</th>
<th colspan="2" halign="left">d</th>
</tr>
<tr>
<th>e</th>
<th>f</th>
<th>...</th>
<th>e</th>
<th>f</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>1</td>
<td>...</td>
<td>6</td>
<td>7</td>
</tr>
<tr>
<td>8</td>
<td>9</td>
<td>...</td>
<td>14</td>
<td>15</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
<tr>
<td>48</td>
<td>49</td>
<td>...</td>
<td>54</td>
<td>55</td>
</tr>
<tr>
<td>56</td>
<td>57</td>
<td>...</td>
<td>62</td>
<td>63</td>
</tr>
</tbody>
</table>
Loading