Skip to content

BUG-22984 Fix truncation of DataFrame representations #22987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 124 commits into from
Nov 15, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
eb4a239
BUG-22984 Fix truncation of DataFrame representations
JustinZhengBC Oct 4, 2018
8e82c82
BUG-22984 Fix flake8 issues
JustinZhengBC Oct 4, 2018
448153d
BUG-22984 Fix whatsnew and add test
JustinZhengBC Oct 6, 2018
244b295
BUG-22984 Fix whatsnew and add test
JustinZhengBC Oct 6, 2018
aa867b0
BUG-22984 Fix linting issue
JustinZhengBC Oct 7, 2018
34b464f
DOC: Updating the docstring of Series.dot (#22890)
HubertKl Nov 3, 2018
4629504
Fixing flake8 problems new to flake8 3.6.0 (#23472)
datapythonista Nov 3, 2018
696e8c7
strictness and checks for Timedelta _simple_new (#23433)
jbrockmendel Nov 3, 2018
3faf1a9
REF: cython cleanup, typing, optimizations (#23464)
jbrockmendel Nov 3, 2018
126edd9
ENH: Add FrozenList.union and .difference (#23394)
gfyoung Nov 3, 2018
da08eeb
BUG: Allow freq conversion from dt64 to period (#23460)
TomAugspurger Nov 3, 2018
29239ad
add number of Errors, Warnings to scripts/validate_docstrings.py (#23…
hongshaoyang Nov 3, 2018
90be7b3
DOC: Add cookbook entry for triangular correlation matrix (GH22840) (…
Nov 3, 2018
defff22
CLN: doc string (#23469)
topper-123 Nov 3, 2018
7aed9e6
STYLE: Standardize cython spacing for casting, with linting (#23474)
jbrockmendel Nov 4, 2018
f1768c7
DOC: Adding documentation for pandas.core.indexes.api internal functi…
Nov 4, 2018
547c240
DOC: Clarify documentation of 'ambiguous' parameter (#23408)
bartaelterman Nov 4, 2018
cb51a02
DOC: Validate in docstrings that numpy and pandas are not imported (#…
thoo Nov 4, 2018
ae938fd
DOC: Updated docstrings related to DateTimeIndex. GH22459 (#22504)
eldritchideen Nov 4, 2018
8c29ede
DOC: Rephrased doc for Series.asof. Added examples (#21034)
dragosthealex Nov 4, 2018
b0c4156
TST: drop tz-aware timestamp from DatetimIndex with DST transition (#…
mroeschke Nov 4, 2018
23fae32
DOC: update the pandas.Series.shift docstring (#20472)
ZackStone Nov 4, 2018
1f4bad5
Isort contributing guide (#23364)
alimcmaster1 Nov 4, 2018
e938ed4
API: fix corner case of lib.infer_dtype (#23422)
h-vetinari Nov 4, 2018
39a2e0a
DOC: Use flake8 to check for PEP8 violations in doctests (#23399)
FHaase Nov 4, 2018
1b00ec5
TST: Empty Series.reindex with tz-aware dtype (#23480)
mroeschke Nov 4, 2018
0d0b561
DOC: Fix creation of [source] links in the doc creation (#23129)
vkk800 Nov 4, 2018
6202eb0
DOC: Remove dead link and update links to https (#23476)
mroeschke Nov 5, 2018
58dc180
DOC: Fix DataFrame.nlargest and DataFrame.nsmallest doctests (#23202)
Moisan Nov 5, 2018
55e2f3d
DOC: Fix syntax error in groupby docs (#23498)
gfyoung Nov 5, 2018
5e09698
Run Isort on tests-> util,sereis,arrays (#23501)
thoo Nov 5, 2018
ed46d6d
TST: Add test of assignment chaining and dupe cols (#23487)
gfyoung Nov 5, 2018
c3f6b8b
BUG: Avoid casting to double type unnecessarily when setting values i…
icexelloss Nov 5, 2018
9a77c2f
Bumping up min version for pyarrow and fastparquet (#23482)
anjsudh Nov 5, 2018
feede99
CI: Fixed pytest minversion (#23520)
TomAugspurger Nov 6, 2018
32c22d4
CI: Exclude asv benchmark envs from flake8 checks (#23497)
qwhelan Nov 6, 2018
b34cfff
Extraneous parts broken off from other PRS (#23518)
jbrockmendel Nov 6, 2018
53dad83
Fixing typo in cython casting lint, and making it azure friendly (#23…
datapythonista Nov 6, 2018
510ba4e
CI: Unpin NumPy (#23465)
TomAugspurger Nov 6, 2018
6a88f0e
TST: Add test for mangling of unnamed columns (#23485)
gfyoung Nov 6, 2018
37dd36f
PERF: cython optimizations (#23477)
jbrockmendel Nov 6, 2018
2b626d5
BUG: Cleanup timedelta offset (#23439)
sinhrks Nov 6, 2018
9436e21
API: change default for sep in str.cat (in docstring) (#23443)
h-vetinari Nov 6, 2018
1071c28
BUG: fix df.where(cond) when cond is empty (#21947)
pajachiet Nov 6, 2018
079f632
BUG: Fix of handle missing CSV MI column names (#23484)
gfyoung Nov 6, 2018
013315a
BUG GH23451 Allow setting date in string index for Series (#23495)
JustinZhengBC Nov 6, 2018
662759a
BUG: fix groupby.transform rename bug (#23461) (#23463)
DiegoAlbertoTorres Nov 6, 2018
6efd331
BUG: GroupBy return EA dtype (#23318)
5hirish Nov 6, 2018
1da1d63
BUG - pd.concat with all Series on axis=1 ignores the `names` argumen…
ms7463 Nov 6, 2018
2acb22c
BUG: names on union and intersection for Index were inconsistent (#19…
Dr-Irv Nov 6, 2018
898c2a2
DOC/TST: Fix warning in concat docstring (#23529)
TomAugspurger Nov 6, 2018
fa8e130
CI: Moving CI dependencies to "ci/deps" (#23526)
datapythonista Nov 6, 2018
278124e
Fix+test timedelta64(nat) ops (#23425)
jbrockmendel Nov 6, 2018
863e628
CLN: Isort io and compat dirs (#23534)
alimcmaster1 Nov 7, 2018
5244647
CI: Fail Azure CI for failing tests (#23454)
alimcmaster1 Nov 7, 2018
c611272
Make validate_docstrings.py ready for the CI (#23514)
datapythonista Nov 7, 2018
4d4b583
Fixing bug in validate_docstrings.py, where a bracket closed in the w…
datapythonista Nov 7, 2018
8c280ca
BUG: Timestamp retains frequency of input Timestamps (#23503)
mroeschke Nov 7, 2018
574a03a
BUG: DatetimeIndex slicing with boolean Index raises TypeError (#22852)
sinhrks Nov 7, 2018
73cc01b
BUG: Return KeyError for invalid string key (#23540)
toobaz Nov 7, 2018
43b135f
TST: coverage for skipped tests in io/formats/test_to_html.py (#22888)
simonjayhawkins Nov 7, 2018
31eee47
DOC: Updating DataFrame.join docstring (#23471)
datapythonista Nov 7, 2018
737cb7d
ENH: Support EAs in Series.unstack (#23284)
TomAugspurger Nov 7, 2018
13f41ae
BUG: Fix error message for invalid HTML flavor (#23550)
gfyoung Nov 8, 2018
6d4178a
remove uses of (ts)?lib.(NaT|iNaT|Timestamp) (#23562)
jbrockmendel Nov 8, 2018
66a95bc
BUG: raise if invalid freq is passed (#23546)
toobaz Nov 8, 2018
3bb2f75
TST: Fix dtype mismatch on 32bit in IntervalTree get_indexer test (#2…
jschendel Nov 8, 2018
af33308
Preserve EA dtype in DataFrame.stack (#23285)
TomAugspurger Nov 8, 2018
cf5e385
CI: Auto-cancel redundant builds (#23523)
TomAugspurger Nov 8, 2018
7c0b5d0
ENH: Support writing timestamps with timezones with to_sql (#22654)
mroeschke Nov 8, 2018
db58d3d
DOC: Added note about groupby excluding Decimal columns by default (#…
Nov 8, 2018
b62634f
Fixes to make validate_docstrings.py not generate warnings or unwante…
datapythonista Nov 8, 2018
2c193a0
Update description of Index._values/values/ndarray_values (#23507)
jorisvandenbossche Nov 8, 2018
6244f35
TST: Tests and Helpers for Datetime/Period Arrays (#23502)
jbrockmendel Nov 9, 2018
03d632c
PERF: define is_all_dates to shortcut inadvertent copy when slicing a…
qwhelan Nov 9, 2018
01ffb03
PERF: Speeds up creation of Period, PeriodArray, with Offset freq (#2…
TomAugspurger Nov 9, 2018
56b8024
DOC/CI: Add linting to rst files, and fix issues (#23381)
FHaase Nov 9, 2018
84ef701
CLN: remove values attribute from datetimelike EAs (#23603)
jorisvandenbossche Nov 9, 2018
e4104e9
TST: Use intp as expected dtype in IntervalIndex indexing tests (#23609)
jschendel Nov 10, 2018
eefb76e
ENH: Support for partition_cols in to_parquet (#23321)
anjsudh Nov 10, 2018
af8ca26
DOC: Updating Series.resample and DataFrame.resample docstrings (#23197)
jmrr Nov 10, 2018
d8d6222
MAINT: tm.assert_raises_regex --> pytest.raises (#23592)
gfyoung Nov 10, 2018
cffdb0e
DOC: Remove incorrect periods at the end of parameter types (#23600)
meiermark Nov 11, 2018
b265308
DOC: Fixes to docstring to add validation to CI (#23560)
datapythonista Nov 11, 2018
f2eac44
TST: Fix integer ops comparison test (#23619)
TomAugspurger Nov 11, 2018
fef6d7a
TST: Unskip some Categorical Tests (#23613)
TomAugspurger Nov 11, 2018
c80ff12
DOC: Fix Order of parameters in docstrings (#23611)
thoo Nov 11, 2018
ff8130b
CLN: use float64_t consistently instead of double, double_t (#23583)
jbrockmendel Nov 11, 2018
9977a08
API: DataFrame.__getitem__ returns Series for sparse column (#23561)
TomAugspurger Nov 11, 2018
1d861de
BUG: Delegate more of Excel parsing to CSV (#23544)
gfyoung Nov 11, 2018
c355f26
BUG: Casting tz-aware DatetimeIndex to object-dtype ndarray/Index (#2…
jbrockmendel Nov 11, 2018
a4c1490
BUG/REF: TimedeltaIndex.__new__ (#23539)
jbrockmendel Nov 11, 2018
468df9c
BUILD: Simplifying contributor dependencies (#23522)
datapythonista Nov 11, 2018
6401a4f
TST: Fix xfailing DataFrame arithmetic tests by transposing (#23620)
jbrockmendel Nov 12, 2018
92b015d
DOC: Enhancing pivot / reshape docs (#21038)
VincentLa Nov 12, 2018
519de0b
CLN:Remove unused **kwargs from user facing methods (#23249)
kprestel Nov 12, 2018
d4be020
CI: Check in the CI that assert_raises_regex is not being used (#23627)
datapythonista Nov 12, 2018
d8826bf
CLN: datetimelike arrays: isort, small reorg (#23587)
jbrockmendel Nov 12, 2018
3b87703
BUG: astype fill_value for SparseArray.astype (#23547)
TomAugspurger Nov 12, 2018
999ef43
More helpful Stata string length error. (#23629)
jtkiley Nov 12, 2018
7fc3732
DEPR: Deprecate usecols as int in read_excel (#23635)
gfyoung Nov 12, 2018
bb9f4eb
BUG: Don't over-optimize memory with jagged CSV (#23527)
gfyoung Nov 12, 2018
76c078c
DOC: Adding validation of the section order in docstrings (#23607)
datapythonista Nov 12, 2018
fdda5d7
CLN: remove incorrect usages of com.AbstractMethodError (#23625)
jbrockmendel Nov 12, 2018
d3e43f8
DOC: avoid SparseArray.take error (#23637)
TomAugspurger Nov 12, 2018
c7dc40c
DOC: clean-up recent doc errors/warnings (#23636)
jorisvandenbossche Nov 12, 2018
f7556a4
DOC: Fix name of the See Also section titles in docstrings (#23653)
gueletk Nov 13, 2018
951041e
CI: Allow to compile docs with ipython 7.11 #22990 (#23655)
benoxoft Nov 13, 2018
d5d6d91
TST: IntervalTree.get_loc_interval should return platform int (#23660)
jschendel Nov 13, 2018
4242077
CLN: Move to_excel to generic.py (#23656)
gfyoung Nov 13, 2018
c1640c6
Add to_flat_index method to MultiIndex (#22866)
WillAyd Nov 13, 2018
8e4bf4c
BUG: Fix read_excel w/parse_cols & empty dataset (#23661)
gfyoung Nov 13, 2018
b3a3ac7
DOC: Update is_sparse docstring (#19983)
sechilds Nov 13, 2018
7dab45f
DOC: Surface / doc mangle_dupe_cols in read_excel (#23678)
gfyoung Nov 14, 2018
991547e
Fix errorbar visualization (#23674)
justinchan23 Nov 14, 2018
c8ac3bf
DOC: Accessing files from a S3 bucket. (#23639)
myles Nov 14, 2018
f9563ea
REF: Move Excel names parameter handling to CSV (#23690)
gfyoung Nov 14, 2018
2688cbe
BUG: Fix Series/DataFrame.rank(pct=True) with more than 2**24 rows (#…
jschendel Nov 14, 2018
d0adfb0
CI: raise clone depth limit on CI
jreback Nov 14, 2018
f452c40
Implement _most_ of the EA interface for DTA/TDA (#23643)
jbrockmendel Nov 14, 2018
36f0608
Merge master
JustinZhengBC Nov 14, 2018
8459936
reapply changes
JustinZhengBC Nov 14, 2018
0f7aa4b
r
JustinZhengBC Nov 14, 2018
82fa50c
Merge remote-tracking branch 'upstream/master' into JustinZhengBC-BUG…
TomAugspurger Nov 14, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v0.24.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1319,7 +1319,8 @@ Notice how we now instead output ``np.nan`` itself instead of a stringified form
- :func:`read_sas()` will correctly parse sas7bdat files with many columns (:issue:`22628`)
- :func:`read_sas()` will correctly parse sas7bdat files with data page types having also bit 7 set (so page type is 128 + 256 = 384) (:issue:`16615`)
- Bug in :meth:`detect_client_encoding` where potential ``IOError`` goes unhandled when importing in a mod_wsgi process due to restricted access to stdout. (:issue:`21552`)
- Bug in :func:`to_string()` that broke column alignment when ``index=False`` and width of first column's values is greater than the width of first column's header (:issue:`16839`, :issue:`13032`)
- Bug in :func:`DataFrame.to_string()` that broke column alignment when ``index=False`` and width of first column's values is greater than the width of first column's header (:issue:`16839`, :issue:`13032`)
- Bug in :func:`DataFrame.to_string()` that caused representations of :class:`DataFrame` to not take up the whole window (:issue:`22984`)
- Bug in :func:`DataFrame.to_csv` where a single level MultiIndex incorrectly wrote a tuple. Now just the value of the index is written (:issue:`19589`).
- Bug in :meth:`HDFStore.append` when appending a :class:`DataFrame` with an empty string column and ``min_itemsize`` < 8 (:issue:`12242`)
- Bug in :meth:`read_csv()` in which :class:`MultiIndex` index names were being improperly handled in the cases when they were not provided (:issue:`23484`)
Expand Down
5 changes: 0 additions & 5 deletions pandas/io/formats/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -608,11 +608,6 @@ def to_string(self):
else: # max_cols == 0. Try to fit frame to terminal
text = self.adj.adjoin(1, *strcols).split('\n')
max_len = Series(text).str.len().max()
headers = [ele[0] for ele in strcols]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a number of tests for this, suprised this didn't break anything, can you make a test if we have no coverage for this case now? (e.g. you use option_context to set the width, then check the output string)

# Size of last col determines dot col size. See
# `self._to_str_columns
size_tr_col = len(headers[self.tr_size_col])
max_len += size_tr_col # Need to make space for largest row
# plus truncate dot col
dif = max_len - self.w
# '+ 1' to avoid too wide repr (GH PR #17023)
Expand Down
17 changes: 12 additions & 5 deletions pandas/tests/io/formats/test_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -305,14 +305,10 @@ def test_repr_non_interactive(self):
assert not has_truncated_repr(df)
assert not has_expanded_repr(df)

def test_repr_truncates_terminal_size(self):
def test_repr_truncates_terminal_size(self, mock):
# https://github.com/pandas-dev/pandas/issues/21180
# TODO: use mock fixutre.
# This is being backported, so doing it directly here.
try:
from unittest import mock
except ImportError:
mock = pytest.importorskip("mock")

terminal_size = (118, 96)
p1 = mock.patch('pandas.io.formats.console.get_terminal_size',
Expand Down Expand Up @@ -343,6 +339,17 @@ def test_repr_truncates_terminal_size(self):

assert df2.columns[0] in result.split('\n')[0]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: we have a pytest fixture formock now, that does the try / except / skip done above.

I think it'd be cleanest to split this into a new test here, and accept the mock parameter.

def test_repr_truncates_terminal_size_full(self, mock):
    ...

Any if you're feeling adventurous, you could change the try / except / skip mock import above to use the fixture as well. Not a big deal though.

def test_repr_truncates_terminal_size_full(self, mock):
# GH 22984 ensure entire window is filled
terminal_size = (80, 24)
df = pd.DataFrame(np.random.rand(1, 7))
p1 = mock.patch('pandas.io.formats.console.get_terminal_size',
return_value=terminal_size)
p2 = mock.patch('pandas.io.formats.format.get_terminal_size',
return_value=terminal_size)
with p1, p2:
assert "..." not in str(df)

def test_repr_max_columns_max_rows(self):
term_width, term_height = get_terminal_size()
if term_width < 10 or term_height < 10:
Expand Down