ENH: add math mode to formatter escape="latex-math" #50398

natmokval · 2022-12-22T16:43:20Z

closes ENH: add math mode to formatter escape="latex" #50040
All code checks passed.
Tests added and passed.

Added latex-math mode to avoid escaping $.
We probably shouldn't escape\[ and \]. They can not be applied between \begin{tabular} and \end{tabular}, which is used for both Series and DataFrames.

natmokval · 2022-12-29T19:25:40Z

Hi @attack68, could you please review this PR?

attack68

The escape option is also a global pandas option configurable. We should expand the documentation there to specify the new option also.

A question: the characters _, {, }, ^ and \ are frequently used within the math mode, i.e. between $'s to specify equations. Does preserving the $'s but still escaping those math-related characters inside $'s break the representation of the math in Latex?

For example what happens to $ \alpha = \frac{\beta}{\zeta^2} $ under this escape mode?

attack68 · 2022-12-30T07:28:55Z

pandas/io/formats/style_render.py

+            Use 'latex-math' to replace the characters ``&``, ``%``, ``#``, ``_``,
+            ``{``, ``}``, ``~``, ``^``, and ``\``
+            in the cell display string with LaTeX-safe sequences.


Whilst this is true I think it is clearer to say:

"Replaces the same characters as the latex option except $ which is preserved to allow conversion to math mode".

We should also add an example probably.

attack68 · 2022-12-30T07:29:52Z

pandas/io/formats/style_render.py

+def _escape_latex_math(s):
+    r"""
+    Replace the characters ``&``, ``%``, ``#``, ``_``, ``{``, ``}``,
+    ``~``, ``^``, and ``\`` in the string with LaTeX-safe sequences.


I would add here: note ``$`` not escaped.

natmokval · 2023-01-19T22:12:40Z

Hi @attack68, I checked the example $ \alpha = \frac{\beta}{\zeta^2} $ , and it didn’t work with the original fix. So I created more LaTeX expressions in math mode to understand how math-mode should work. My suggestion: there is no easy way to escape characters in math-mode. We can rely on user, he knows how to deal with LaTeX expressions.

For example: On one hand, the characters { and } are used in expressions to select logically a part of the expression. On the other hand, \{ and \} are curly brackets. We cannot decide for the user what he meant.

I added an example for math-mode and corrected LaTeX mode example.

attack68 · 2023-01-20T10:05:53Z

I think we may be able to use regex to detect these. Im not brialliant at regex, but doing a search on stack overflow we can use look back and look forward, I think. These would be the rules I thiunk we aim for:

all of the usual characters, except $, are escaped, unless:
any of these charcaters are detected as being between a left side $ and a right side $, in which case they form part of the equation and should not be escaped.

natmokval · 2023-01-20T11:51:07Z

I think we may be able to use regex to detect these. Im not brialliant at regex, but doing a search on stack overflow we can use look back and look forward, I think. These would be the rules I thiunk we aim for:
* all of the usual characters, except `$`, are escaped, **unless**:

* any of these charcaters are detected as being between a left side `$` and a right side `$`, in which case they form part of the equation and should not be escaped.

Thank you, I understand. I'll try to use regex.

natmokval · 2023-01-26T21:29:54Z

Hi @attack68. As you suggested I made a solution using regex to detect math-mode. It works well for expressions that are surrounded by the character $. For example: "abc $first\$math$ middle $secondmath$"

On the other hand, math expressions can be surrounded by the characters $ and $.
For example: " ab $firstmath$ middle $ secondmath $ %(x)^ "
Or even with some mistakes, like " \)ab $firstmath$ middle $ secondmath $ %(x)^ \)"
For this case, I have an approach as well.

I was thinking about updating the PR but I still have one question. Should we consider the case when we have an expression containing both math-mode types, $ and $`` $? For example: "abc $firstmath$ middle $secondmath$"
It doesn’t look easy to process such expression.

attack68 · 2023-01-26T21:51:10Z

I think a basic PR just to handle and test the $ math mode case will be a good start. Once that is in place we could extend it in the future, and with the tests developed provides a reliable base from which to work

natmokval · 2023-01-29T15:31:48Z

Hi @attack68. I added a basic implementation for math mode between $ and a test.
Could you please take a look?

attack68 · 2023-01-30T10:05:37Z

pandas/io/formats/style_render.py

+    str :
+        Escaped string
+    """
+    s = s.replace(r"\$", r"ab2§=§8yz")


I think I understand what you are doing here, you are substituting out an escaped $ sign, using a proxy uuid string ab2§=§8yz, but this string is the same string as I have used in the _escape_latex method for the same reason for dealing with some other character combiantions. I wonder if this might cause cross-pollution. Regardless, since it is a procy uuid string, it would be better to change the digits I think (using only digits that are not escaped!)

attack68 · 2023-01-30T10:09:54Z

pandas/io/formats/style_render.py


-        >>> df = pd.DataFrame([["123"], ["~ ^"], ["$%#"]])
+        >>> df = pd.DataFrame([["123"], ["~ ^"], ["%#"]])


Its fine to change this example. Just curious as to the reason.

I think one change worth making, otherwise I think it is good to add it. The test demonstrates the effect nicely. Didn't check the CI tests, we make sur ethey are green before merging.

I changed uuid string to be unique, as you suggested. And also reverted the example for latex mode. I changed it by mistake, there is nothing wrong with $ in latex mode (the online latex compiler which I used at first had some sort of bug).

CI tests look good.

attack68

I think one change worth making, otherwise I think it is good to add it.
The test demonstrates the effect nicely.
Didn't check the CI tests, we make sur ethey are green before merging.

attack68 · 2023-02-10T18:41:50Z

Ah I just noticed there is no addition to the whatsnew. You might to add a simple one line addition.

natmokval · 2023-02-11T14:47:13Z

Thank you, I added a line to whatsnew.

attack68

LGTM, thanks.

mroeschke · 2023-02-13T20:55:10Z

Could you merge in main and resolve the merge conflict?

natmokval · 2023-02-13T21:45:31Z

I merged main in and resolved the merge conflict.

natmokval · 2023-03-04T12:20:00Z

I resolved a new merge conflict and did merge in main. Sorry, it took so long.
Is there anything else that is needed to be done for this PR?

attack68 · 2023-03-05T14:45:33Z

@mroeschke can we still get this in to 2.0?

mroeschke

Okay sure. This is okay for 2.0 since it looks to have been ready for a while and started before the RC release

mroeschke · 2023-03-06T08:34:23Z

Thanks @natmokval

natmokval · 2023-03-06T12:50:52Z

Thank you for reviewing my PR.
I thought it might be useful to add to latex mode a case when a math expression starts with $ and ends with $. I'd like to do it and open a new PR.

attack68 · 2023-03-06T13:18:52Z

I think you can use all of the following:

These are more difficult to cover I think. I'm glad we have this PR to get a basic latex-math mode in there but these will require extensive tests I think.
A new PR will probably have to be quite test driven I would expect.

I dont know which of these tend to display in Mathjax either, for notebook display.

natmokval marked this pull request as draft December 23, 2022 10:42

natmokval force-pushed the 50040-add-math-mode-formatter-escape=latex branch 2 times, most recently from 4fb4d86 to adedc02 Compare December 29, 2022 15:31

natmokval marked this pull request as ready for review December 29, 2022 19:23

natmokval requested a review from attack68 December 29, 2022 19:24

attack68 requested changes Dec 30, 2022

View reviewed changes

attack68 added IO LaTeX to_latex Styler conditional formatting using DataFrame.style Enhancement labels Dec 31, 2022

ENH: add escape math mode with escape=latex-math

4f9e8e1

natmokval force-pushed the 50040-add-math-mode-formatter-escape=latex branch from 8a329f8 to 4f9e8e1 Compare January 28, 2023 21:20

ENH: add test for escape=latex-math

43800a4

attack68 reviewed Jan 30, 2023

View reviewed changes

attack68 requested changes Jan 30, 2023

View reviewed changes

change uuid string and revert the example for latex

2fd6d7f

natmokval and others added 2 commits February 11, 2023 15:36

add one line to whatsnew

f80a1c4

Merge branch 'main' into 50040-add-math-mode-formatter-escape=latex

1c268e8

attack68 approved these changes Feb 13, 2023

View reviewed changes

Merge branch 'main' into 50040-add-math-mode-formatter-escape=latex

ad777d1

ENH: add math mode to formatter escape=latex-math IV

e80ffbb

mroeschke approved these changes Mar 6, 2023

View reviewed changes

mroeschke merged commit dfd5d66 into pandas-dev:main Mar 6, 2023

natmokval mentioned this pull request Mar 11, 2023

ENH: add LaTeX math mode with parentheses #51903

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add math mode to formatter escape="latex-math" #50398

ENH: add math mode to formatter escape="latex-math" #50398

natmokval commented Dec 22, 2022 •

edited

Loading

natmokval commented Dec 29, 2022

attack68 left a comment •

edited

Loading

attack68 Dec 30, 2022

attack68 Dec 30, 2022

natmokval commented Jan 19, 2023

attack68 commented Jan 20, 2023

natmokval commented Jan 20, 2023

natmokval commented Jan 26, 2023

attack68 commented Jan 26, 2023

natmokval commented Jan 29, 2023

attack68 Jan 30, 2023

attack68 Jan 30, 2023

natmokval Jan 31, 2023

attack68 left a comment

attack68 commented Feb 10, 2023

natmokval commented Feb 11, 2023

attack68 left a comment

mroeschke commented Feb 13, 2023

natmokval commented Feb 13, 2023

natmokval commented Mar 4, 2023

attack68 commented Mar 5, 2023

mroeschke left a comment

mroeschke commented Mar 6, 2023

natmokval commented Mar 6, 2023

attack68 commented Mar 6, 2023 •

edited

Loading


		>>> df = pd.DataFrame([["123"], ["~ ^"], ["$%#"]])
		>>> df = pd.DataFrame([["123"], ["~ ^"], ["%#"]])

ENH: add math mode to formatter escape="latex-math" #50398

ENH: add math mode to formatter escape="latex-math" #50398

Conversation

natmokval commented Dec 22, 2022 • edited Loading

natmokval commented Dec 29, 2022

attack68 left a comment • edited Loading

Choose a reason for hiding this comment

attack68 Dec 30, 2022

Choose a reason for hiding this comment

attack68 Dec 30, 2022

Choose a reason for hiding this comment

natmokval commented Jan 19, 2023

attack68 commented Jan 20, 2023

natmokval commented Jan 20, 2023

natmokval commented Jan 26, 2023

attack68 commented Jan 26, 2023

natmokval commented Jan 29, 2023

attack68 Jan 30, 2023

Choose a reason for hiding this comment

attack68 Jan 30, 2023

Choose a reason for hiding this comment

natmokval Jan 31, 2023

Choose a reason for hiding this comment

attack68 left a comment

Choose a reason for hiding this comment

attack68 commented Feb 10, 2023

natmokval commented Feb 11, 2023

attack68 left a comment

Choose a reason for hiding this comment

mroeschke commented Feb 13, 2023

natmokval commented Feb 13, 2023

natmokval commented Mar 4, 2023

attack68 commented Mar 5, 2023

mroeschke left a comment

Choose a reason for hiding this comment

mroeschke commented Mar 6, 2023

natmokval commented Mar 6, 2023

attack68 commented Mar 6, 2023 • edited Loading

natmokval commented Dec 22, 2022 •

edited

Loading

attack68 left a comment •

edited

Loading

attack68 commented Mar 6, 2023 •

edited

Loading