Skip to content

Exception when adding a unicode margins_name in pivot_table #13292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
spapas opened this issue May 26, 2016 · 3 comments · Fixed by #17489
Closed

Exception when adding a unicode margins_name in pivot_table #13292

spapas opened this issue May 26, 2016 · 3 comments · Fixed by #17489
Labels
Error Reporting Incorrect or improved errors from pandas Unicode Unicode strings
Milestone

Comments

@spapas
Copy link

spapas commented May 26, 2016

Code Sample, a copy-pastable example if possible

This throws UnicodeDecodeError in python2

pd.pivot_table(pd.DataFrame({'foo': [1,2,3]}), index=['foo'], aggfunc=len, margins=True, margins_name=u'Δοκιμή')

Expected Output

Not throwing exception!

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 20.10.1
Cython: None
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.2
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext)
jinja2: None
boto: None
pandas_datareader: None

The above is a well-known problem when mixing unicode and non-unicode strings in python2: The fix (for python 2) is very simple, just add a u in front of the exception_msg string literal in https://github.com/pydata/pandas/blob/master/pandas/tools/pivot.py#L178 i.e line 178 should be exception_msg = u'Conflicting name "{0}" in margins'.format(margins_name)

I am not sure what's the correct way to fix that in python 3 and a python2-3 compatible way so I won't provide a PR for that.

@jreback
Copy link
Contributor

jreback commented May 26, 2016

mixing of str with unicode is NEVER recommended.

can check the Index if its inferred_type == 'unicode' if the passed value is also unicode, otherwise could raise an error. So I'll mark this as a better error message.

@jreback jreback added Difficulty Novice Unicode Unicode strings Error Reporting Incorrect or improved errors from pandas labels May 26, 2016
@jreback jreback added this to the Next Major Release milestone May 26, 2016
@jreback jreback changed the title Exception when adding a unicode margins_name in pivot_table Exception when adding a unicode margins_name in pivot_table May 26, 2016
@mpenkov
Copy link
Contributor

mpenkov commented Sep 10, 2017

@sinhrks I'll start with this one, if it's OK.

@mpenkov
Copy link
Contributor

mpenkov commented Sep 10, 2017

I could reproduce the problem. This is the error I'm seeing:

(pandas)bash-3.2$ python
Python 2.7.13 (default, Jun  2 2017, 06:36:47)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.pivot_table(pd.DataFrame({'foo': [1,2,3]}), index=['foo'], aggfunc=len, margins=True, margins_name=u'Δοκιμή')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/reshape/pivot.py", line 126, in pivot_table
    margins_name=margins_name, fill_value=fill_value)
  File "pandas/core/reshape/pivot.py", line 148, in _add_margins
    msg = 'Conflicting name "{name}" in margins'.format(name=margins_name)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Error Reporting Incorrect or improved errors from pandas Unicode Unicode strings
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants