BUG: DataFrame.to_string with formatters, header and index False

[chiroptical]

closes #13032

• tests added / passed - added test specific to format bug
• passes pep8radius master --diff
• whatsnew entry - not needed

Found this bug experimenting with formatters. First pull request to pandas, but I believe guidelines are quite clear. I can explain what was happening in more detail if that is necessary.

[jreback]

is this related to #13032 ?

can you show an example of before / after

[jreback]

add the issue number (this PR since there is no number) as a comment

if you need the import put at the top of the file

[chiroptical]

Turns out is exactly related to #13032. I did see this issue, but didn't make the connection initially. In my testing, the lines should be removed not modified. To be honest, I didn't understand what these lines accomplished when I was reviewing the code originally.

Minimal code to reproduce (stripped down from #13032, easier to see with formatters):

>>> import pandas as pd
>>> frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
>>> formatter = lambda x: '{:10.3f}'.format(x)
>>> print(frame.to_string(index=False, header=False))

Before (adding slashes to make space counting easier):
\ \ \ \ 0.000\ \ \ \ \ \ 0.000
After:
\ \ \ \ \ 0.000\ \ \ \ \ \ 0.000

Output from PR for #13032 (without formatter):

>>> print(df.to_string(index=False))
      one       two     three
 1.722364  0.846757  0.094394
-0.578834  0.836656  0.665414
 0.345460  1.782786  1.760175

Output from PR for #13032 (with formatter 10.3f, slashes added once for column count):

>>> print(df.to_string(index=False,formatters=[formatter,formatter,formatter]))
       one        two      three
\ \ \ \ \ 1.722      0.847      0.094
    -0.579      0.837      0.665
     0.345      1.783      1.760

I will add the PR and Issues numbers plus make corrections shortly.

[codecov-io]

Current coverage is 84.23% (diff: 100%)

Merging #13350 into master will increase coverage by <.01%

@@             master     #13350   diff @@
==========================================
  Files           138        138          
  Lines         50713      50721     +8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42715      42723     +8   
  Misses         7998       7998          
  Partials          0          0          

Powered by Codecov. Last update ce56542...73d7b7e

[chiroptical]

Additionally, the failed Travis CI checks don't appear to be related to my subtrations/additions. Lastly, the unit test must use assert because of the location in the code. Should it be moved?

[jreback]

@barrymoo can you also read thru the comments on that issue (and the referenced one) and see if covering bases.

[chiroptical]

@jreback Of course,

From comment #13032 (comment):
Before:

Name    Value
                                  Short        1
                                 Longer  9374518
Much Longer name to the Max -----------    32432

After (desired output, correct?):

                                    Name    Value
                                   Short        1
                                  Longer  9374518
 Much Longer name to the Max -----------    32432

I think the others comments would be unaffected. To be fair though, I am very new to pandas.

[chiroptical]

I see what you mean now. From #11833:

Code:

>>> import pandas as pd
>>> df = pd.DataFrame({'a':range(5)})
>>> df.to_string(index=False)
u' a\n 0\n 1\n 2\n 3\n 4'
>>> formatter = lambda x: '{:1d}'.format(x)
>>> df.to_string(formatters=[formatter], index=False)
u'a\n0\n1\n2\n3\n4'

I should be able to fix this issue quickly.

[chiroptical]

A different perspective: where are those spaces coming from in the first place? (I will try to track this down)

[chiroptical]

Code:

#!/usr/bin/env python
import pandas as pd
import numpy as np

# Test 1
frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
formatter = lambda x: '{:10.3f}'.format(x)
string = frame.to_string(index=False, header=False)
print('--> Begin Test 1 <--')
print(string)
print('-->  End Test 1  <--')

# Test 2
df = pd.DataFrame(np.random.randn(3, 3), columns=['one', 'two', 'three'])
string = df.to_string(index=False)
print('--> Begin Test 2 <--')
print(string)
print('-->  End Test 2  <--')

# Test 3
df = pd.DataFrame({'a':range(5)})
string = df.to_string(index=False)
print('--> Begin Test 3 <--')
print(string)
print('-->  End Test 3  <--')

# Test 4
NAMES = ['Short', 'Longer', 'Much Longer name to the Max -----------']
VALUES = [1, 9374518, 32432]
d = pd.DataFrame({'Name': NAMES, 'Value': VALUES})
string = d.to_string(index=False)
print('--> Begin Test 4 <--')
print(string)
print('-->  End Test 4  <--')

Produces:

$ python quick-test.py
--> Begin Test 1 <--
0 0
-->  End Test 1  <--
--> Begin Test 2 <--
      one       two     three
-0.117275 -0.410192 -2.170441
 0.194766  0.521318  0.936951
-0.923841 -1.829388  1.078478
-->  End Test 2  <--
--> Begin Test 3 <--
a
0
1
2
3
4
-->  End Test 3  <--
--> Begin Test 4 <--
                                   Name   Value
                                  Short       1
                                 Longer 9374518
Much Longer name to the Max -----------   32432
-->  End Test 4  <--

I believe this is the desired output. Unfortunately, this code fails about 20 tests. Hopefully, it is because the spacing in the expected output has changed slightly.

[chiroptical]

About 30 failed checks from nosetests pandas/tests/formats/test_format.py. I will checking the expected outputs this weekend. Note, I edited #13350 (comment) to include the new code edits.

[chiroptical]

I have created barrymoo/pandas-pr-13350-supplement to document the test failures (and eventually generate new expected strings). This is a work-in-progress.

[chiroptical]

I have added some more tests to the supplement. There is one test which I am having some difficulty with, starts: https://github.com/barrymoo/pandas-pr-13350-supplement/blob/master/tests.py#L474. I think the frame is supposed to overcome the terminal size, but it doesn't.

[chiroptical]

Hey @jreback I finished my supplement but I need some opinions about output formatting especially concerning the test_*_east_asian* tests (I don't know what these should look like). Is there anyone else we could pull in to review this? That way I can fix the rest of the formatting concerns and fix everything with one PR.

For the supplement, clone it, activate the dev environment (tested with 2 & 3, but have not examined diffs of the output), run python tests.py, and review formatting. Submit issues to the other repo with concerns of specific tests.

[jreback]

@barrymoo not sure what you mean by supplement. simply update this PR, comments can just be done here.
cc @sinhrks

[sinhrks]

@barrymoo I live in Japan and am willing to check the output format amd provide test cases:)

[chiroptical]

@jreback I ripped out the failing tests so one can easily print the results out on the command line. That way I can get some input from the community on how people want things formatted and make additional changes.

[chiroptical]

Here's a great example for why I need the supplement. For test_datetimelike_frame, my changes lead to the following output

                          dt  x
0  2011-01-01 00:00:00-05:00  1
1  2011-01-01 00:00:00-05:00  2
..                       ... ..
8                        NaT  9
9                        NaT 10

But, do you like:

                          dt  x
 0 2011-01-01 00:00:00-05:00  1
 1 2011-01-01 00:00:00-05:00  2
..                       ... ..
 8                       NaT  9
 9                       NaT 10

or...

                         dt  x
0 2011-01-01 00:00:00-05:00  1
1 2011-01-01 00:00:00-05:00  2
.                       ... ..
8                       NaT  9
9                       NaT 10

I can easily generate all of these outputs. Or, would you rather I pick what I like and get all of the tests working.

[evanpw]

I also worked on this; getting all of the tests to pass afterward is a nightmare. It looks like this change removes the leading space on integers but leaves it on floats. Is that true?

[chiroptical]

@evanpw it's very time consuming. I am still working through all of the tests, but I don't have a ton of free time. If you're looking at all positive numbers there is an extra space for the nonexistent "-" sign. There is an example in one of the above comments.

[evanpw]

After this change there won't be an extra leading space for a column of positive integers, but there will still be one for a column of positive floats, right?

[chiroptical]

That's correct, but I can likely fix that too. Again the majority of this work is fixing the tests.

[jreback]

can you rebase / update?

[chiroptical]

I didn't do this correct, sorry have not mastered this bit of git yet. I will submit a different pull request