Escape special characters in to_latex() output

[bjornarneson]

Some characters have special meaning in latex: & % $ # _ { } ~ ^ . When a dataframe contains a special character (as part of an index label, for example), to_latex() spits out invalid latex.

[bjornarneson]

May consider escaping all of these:

& % $ # _ { } ~ ^ \

Pandas already deals with three of them:

% & _

[jtratner]

Looks fine, need a minimal test case to ensure this behavior comes out
correctly (i.e., a few rows, covering all the characters you mention and
then make sure that everything was escaped appropriately).

[cancan101]

Since all of the escaping has the same form (append ), maybe consider writing a function that you pass a list of symbols to?

[jreback]

can you add some tests for this?

[jreback]

@y-p hows this look?

[ghost]

Looks ok to me.

[jreback]

@bjornarneson can you squash this down to a single commit?

see this: https://github.com/pydata/pandas/wiki/Using-Git

[jreback]

@bjornarneson ...look ok and passes...ready to merge?

[jreback]

actually...can you add a release note that indicates this change? (in 0.13.1 improvements would be fine, ref this PR as the issue)

[jreback]

pls rebase this on master and squash down to a single commit....see here:https://github.com/pydata/pandas/wiki/Using-Git

[bjornarneson]

@jreback It is probably obvious that I know only enough Git to be dangerous.... I have tried to follow the directions at the link you provided, but have been stymied by failed merges/pushes/rebases/etc. Short of starting over completely, fixing the mess I have made is beyond my current abilities right now. I am sorry about this!

[TomAugspurger]

Someone can walk you through it; git can be frustrating at first (and once you're familiar with it, come to think of it). Checkout your patch-1 branch and do a git status to start and report back what you see.

[bjornarneson]

Thanks a lot -- I'd like to learn how to do this properly. I checked out a clean copy of my patch-1 branch:

bjorna@9722-HP9470 ~/Documents/GitHub/pandas (patch-1)
$ git status
# On branch patch-1
nothing to commit, working directory clean

Then I fetch/rebase:

git fetch upstream
git rebase --interactive upstream/master

...and use the interactive window to squash all the commits into one commit. Then I get this message:

bjorna@9722-HP9470 ~/Documents/GitHub/pandas (patch-1)
$ git rebase --interactive upstream/master
error: could not apply 34d700a... Escape special characters when writing datafra
me to latex

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
Could not apply 34d700a... Escape special characters when writing dataframe to l
atex

Git status shows this:

bjorna@9722-HP9470 ~/Documents/GitHub/pandas (patch-1|REBASE-i 1/6)
$ git status
# rebase in progress; onto d7c6801
# You are currently rebasing branch 'patch-1' on 'd7c6801'.
#   (fix conflicts and then run "git rebase --continue")
#   (use "git rebase --skip" to skip this patch)
#   (use "git rebase --abort" to check out the original branch)
#
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   pandas/core/format.py
#       modified:   pandas/tests/test_format.py
#
# Unmerged paths:
#   (use "git reset HEAD <file>..." to unstage)
#   (use "git add <file>..." to mark resolution)
#
#       both modified:      doc/source/release.rst
#

This is the point where I don't know how to proceed. Any teaching would be most appreciated.

[TomAugspurger]

That last output is saying that git can't automatically merge your changes to release.rst and what's been done on master since you made your changes. If you open up that file, doc/source/release.rst, you'll see that git inserted some stuff where the two branches overlap:

<<<<<<< HEAD

# some text here from head

=======
  - escape special characters when writing to latex (:issue: `5374`)
>>>>>>> 34d700a... Escape special characters when writing dataframe to latex

Delete the <<<<<<< HEAD, =======, and >>>>>>> 34d700a... Escape special characters when writing dataframe to latex and move your - escape special characters when writing to latex (:issue:5374) line just above the .. _release.bug_fixes-0.13.1: line.

Then add that file git add doc/source/release.rst to say you've fixed the conflicts, git rebase --continue and then git push -f origin to force push your changes.

[bjornarneson]

Thanks Tom -- I appreciate the help. Hopefully this last commit is in good shape. I think I was fouling things up with another command somewhere between manually fixing the release.rst file and the force push.

[jreback]

@bjornarneson thanks!

and @TomAugspurger thanks for the help!

[vfilimonov]

Hi guys. Are there any way to switch off escaping? I.e. if there's a column with the latex-formatted text:

>>> tmp = pd.DataFrame({'a':[10,20,30], 'b':['$1\pm0.1$','$2\pm0.2$','$3\pm0.3$']})
>>> print tmp.to_latex()
\begin{tabular}{lrl}
\toprule
{} &   a &          b \\
\midrule
0 &  10 &  \$1\textbackslashpm0.1\$ \\
1 &  20 &  \$2\textbackslashpm0.2\$ \\
2 &  30 &  \$3\textbackslashpm0.3\$ \\
\bottomrule
\end{tabular}

formatters do not help in this situation:

print tmp.to_latex(formatters={'a':lambda x: '%d'%(x), 'b':lambda x: '%s'%(x)})

will result in the same output.

[dalcolmo]

I have also been bitten by this, as I use latex formatting and even latex macros inside strings which are then to be rendered with to_latex:
The symbol lambda in the following table will render correctly in an ipython notebook, but not when using to_latex to render it (paste this into an ipython notebook, then use nbconvert --to-latex and try to run pdflatex on the result and you will see what I mean)

import pandas as pd # version 0.13 revno 5096 or later
pd.DataFrame._repr_latex_ = pd.DataFrame.to_latex
df = pd.DataFrame(index=['value'])
df['$\lambda$'] = '3.14'
df.T

[dalcolmo]

What I wanted to say with my former comment, I also need an option to switch off escaping of latex characters!

[jreback]

easy enough to do this

see #6472
@dalcolmo @vfilimonov one of you want to do a PR?

simply add a escape='True' keyword to maintain compat, and then you can just turn it off....