BUG: lzma is a required part of python, make it optional

[guilherme-salome]

Importing lzma when Python has been compiled without its support (usually due to a lack of xz in the system) will raise a warning. Substituted import lzma for helper function.

• closes import pandas error for missing compression libraries #27575
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[guilherme-salome]

I cannot modify the pandas/_libs/parsers.pyx without running into issues with cythonize (via pip install -e .).

[TomAugspurger]

Thanks @salompas.

We'll also want

1. A release note in 0.25.1.rst. Can go under the "IO" bug fixes section.
2. Update the uses of lzma to raise a RuntimeError when a broken Python install tries to use lzma
3. A test

For the test, something like

diff --git a/pandas/tests/io/test_compression.py b/pandas/tests/io/test_compression.py
index ce459ab24..0d359dd72 100644
--- a/pandas/tests/io/test_compression.py
+++ b/pandas/tests/io/test_compression.py
@@ -1,5 +1,7 @@
 import contextlib
 import os
+import subprocess
+import textwrap
 import warnings

 import pytest
@@ -125,3 +127,15 @@ def test_compression_warning(compression_only):
         with tm.assert_produces_warning(RuntimeWarning, check_stacklevel=False):
             with f:
                 df.to_csv(f, compression=compression_only)
+
+
+def test_with_missing_lzma():
+    # https://github.com/pandas-dev/pandas/issues/27575
+    code = textwrap.dedent(
+        """\
+    import sys
+    sys.modules['lzma'] = None
+    import pandas
+    """
+    )
+    subprocess.check_output(["python", "-c", code])

[TomAugspurger]

I cannot modify the pandas/_libs/parsers.pyx without running into issues with cythonize (via pip install -e .).

What issues do you see? You'll need a C compiler, since you're updating a Cython file. Typically I use python setup.py build_ext --inplace, but pip install -e . may do the same.

[guilherme-salome]

I cannot modify the pandas/_libs/parsers.pyx without running into issues with cythonize (via pip install -e .).

What issues do you see? You'll need a C compiler, since you're updating a Cython file. Typically I use python setup.py build_ext --inplace, but pip install -e . may do the same.

I was adding an if lzma is None to one of the lines to see if RuntimeError is necessary.
However, when I run pip install -e . after modifying the file, I get:

    ERROR: Complete output from command python setup.py egg_info:
    ERROR: 
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    
    
    cdef extern from "parser/tokenizer.h":
    
        ctypedef enum ParserState:
    	START_RECORD
    ^
    ------------------------------------------------------------
    
    pandas/_libs/parsers.pyx:90:0: Mixed use of tabs and spaces
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/guilhermesalome/Desktop/pandas/setup.py", line 813, in <module>
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/Users/guilhermesalome/Desktop/pandas/setup.py", line 542, in maybe_cythonize
        return cythonize(extensions, *args, **kwargs)
      File "/Users/guilhermesalome/.pyenv/versions/miniconda3-latest/envs/pandas-dev/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
        cythonize_one(*args)
      File "/Users/guilhermesalome/.pyenv/versions/miniconda3-latest/envs/pandas-dev/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1219, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: pandas/_libs/parsers.pyx
    Compiling pandas/_libs/parsers.pyx because it changed.
    [1/1] Cythonizing pandas/_libs/parsers.pyx
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /Users/guilhermesalome/Desktop/pandas/

I am modifying the file in Emacs without any mode for Cython.
The error shows up at a line that has nothing to do with the modification I made.

[guilherme-salome]

I managed to make it work in nano and putting whitespaces by hand. It might be an issue with my Emacs using tabs.

[guilherme-salome]

+def test_with_missing_lzma():
+    # https://github.com/pandas-dev/pandas/issues/27575
+    code = textwrap.dedent(
+        """\
+    import sys
+    sys.modules['lzma'] = None
+    import pandas
+    """
+    )
+    subprocess.check_output(["python", "-c", code])

This is much smarter than what I was doing to test the code (change the name of the lzma.py module).

[guilherme-salome]

I do not understand what the command git diff upstream/master -u -- "*.py" | flake8 --diff does, but if I do git diff master -u -- "*.py" | flake8 --diff" the output is nothing.

[guilherme-salome]

Please let me know if this is looking good, and how I can improve the solution.

[jbrockmendel]

but if I do git diff master -u -- "*.py" | flake8 --diff" the output is nothing

That means it is passing that check.

The isort check is failing. Should be fixed if you run isort -rc pandas

[TomAugspurger]

@salompas it's an isort issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=15881&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba&taskId=e3717827-2a2f-5148-8ad4-7900d2171f6e&lineStart=37&lineEnd=38&colStart=1&colEnd=1

https://dev.pandas.io/development/contributing.html#import-formatting

[guilherme-salome]

@salompas it's an isort issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=15881&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba&taskId=e3717827-2a2f-5148-8ad4-7900d2171f6e&lineStart=37&lineEnd=38&colStart=1&colEnd=1

https://dev.pandas.io/development/contributing.html#import-formatting

Thanks for the link, I will fix the issue.

[guilherme-salome]

I made the changes you suggested, let me know how it is looking now.

[TomAugspurger]

Looks pretty close. Can you

1. Rename import_lzma to _import_lzma (pandas.compat is supposed to be private, but people still use it)
2. move the if lzma is None check to a new method in pandas/compat/__init__.py called _get_lzma_file(lzma). If lzma is None it raises the RuntimeError. Otherwise, it returns lzma.LZMAFile, which we can pass the kwargs to when lzma is not None.
3. Add at least one subprocess-style test that hits the RuntimeError.

[guilherme-salome]

Looks pretty close. Can you

1. Rename import_lzma to _import_lzma (pandas.compat is supposed to be private, but people still use it)
2. move the if lzma is None check to a new method in pandas/compat/__init__.py called _get_lzma_file(lzma). If lzma is None it raises the RuntimeError. Otherwise, it returns lzma.LZMAFile, which we can pass the kwargs to when lzma is not None.
3. Add at least one subprocess-style test that hits the RuntimeError.

Done.
1., 2. - Good idea! Looks way better now.
3. - The test will fail if you do not have lzma, but succeeds otherwise.

[guilherme-salome]

Just for my future reference.
On Emacs if you have the following in your init.el:

(progn (add-hook 'before-save-hook 'whitespace-cleanup)
       (add-hook 'before-save-hook (lambda() (delete-trailing-whitespace))))

When saving a file, the function whitespace-cleanup is called and it substitutes spaces for tabs whenever possible.
To change the behavior so that tabs are substituted for space instead, add the following to init.el:
(setq-default indent-tabs-mode nil)
The variable indent-tabs-mode is used by whitespace-cleanup and will change its behavior.
Now there should not be any issues with pyx files.

[guilherme-salome]

I guess another modification to this code is to show the warning just once (right now it shows up twice), but this might not be worth the time.

[auvipy]

looks great

[TomAugspurger]

I think a double warning is fine. Don't worry about that.

One request for changes on the second test. Otherwise, this looks great.

[guilherme-salome]

@jreback @TomAugspurger
Thanks for the suggestions. I made the changes to the code.
Could you check and let me know where to improve? Thanks!

[TomAugspurger]

Thanks @salompas!

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

$ git checkout 0.25.x
$ git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 ba94f9baaa1a802eee4820a1188db6501a231ae6

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #27882: BUG: lzma is a required part of python, make it optional'

4. Push to a named branch :

git push YOURFORK 0.25.x:auto-backport-of-pr-27882-on-0.25.x

5. Create a PR against branch 0.25.x, I would have named this PR:

"Backport PR #27882 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

[TomAugspurger]

Backported in #28012.

[rth]

I opened #43461 to re-visit this. I would like to propose to remove the import warning if lzma is missing. Please comment in the linked issue if you disagree.