Cleanup compression

[lababidi]

• closes CLN: consolidate compression inference as much as possible #12688
• tests passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry - not needed

[jreback]

we want to remove this statement and its functionaility as we don't support 3.2 anymore

[lababidi]

@jreback how is it possible that i broke the following? I don't think I touched any of these files:

Traceback (most recent call last): File "/Users/travis/build/pydata/pandas/pandas/tests/frame/test_to_csv.py", line 999, in test_to_csv_compression_xz df.to_csv(filename, compression="xz") File "/Users/travis/build/pydata/pandas/pandas/core/frame.py", line 1344, in to_csv formatter.save() File "/Users/travis/build/pydata/pandas/pandas/formats/format.py", line 1551, in save self._save() File "/Users/travis/build/pydata/pandas/pandas/formats/format.py", line 1638, in _save self._save_header() File "/Users/travis/build/pydata/pandas/pandas/formats/format.py", line 1634, in _save_header writer.writerow(encoded_labels) io.UnsupportedOperation: not writable

[jreback]

well that writes to a csv so sure

[lababidi]

But it never calls get_handle??

All the tests pass on my machine. This is strange.

[jreback]

are you sure that test is running? it will skip if you don't have backports.lzma not installed

[lababidi]

yeah, i installed backports.lzma and the test runs on my machine. this is strange. especially since i didn't touch any of the files in the stacktrace.

[jreback]

try creating the exact environment (e.g. create a new environment with ci/requirements-2.7.run/pip)

[lababidi]

how do i install the .run file? pip needs == while the .run file has =

[jreback]

conda install -n pandas --file=ci/requirements.....run
pip install --upgrade -r ci/requirements....pip

[jreback]

can you rebase/update

[jreback]

can you update

[lababidi]

This branch is behind on master. memory_map was added and I need time to understand how that changes the logic.

[jorisvandenbossche]

I think something went wrong with your rebase (lots of commits are included here)

[jorisvandenbossche]

@lababidi Do you have time to update this? (you should start with a rebase, as lots of other commits are not included in this PR)

[jreback]

can you rebase / update?

[lababidi]

@jreback on it now

[lababidi]

@jreback @jorisvandenbossche Again, after rebasing to your code, I find that the tests fail and not due to my changes. This is becoming a detriment to my contributions because it seems someone keeps allowing failing code to get merged. I'm not sure how it happened.

[lababidi]

@jreback @jorisvandenbossche I've given you access to modify this branch, so feel free to modify anything you'd like. Thank you.

[jreback]

are these your changes: https://travis-ci.org/pydata/pandas/jobs/159174397

master has been and is passing continuously. any failures are changes that you made. They can certainly fail other tests, but that just means your changes are not clean.

[jorisvandenbossche]

@lababidi If you look at the test failures in the log @jreback linked to, you see that the failure is in the test_iterators test (in test_parsers.py, so related to this PR).
The test failes at chunks = list(parser), and this parser is a file reader with a PythonParser engine which is passed a list of list of strings instead of a file-like object. Because of your change, this list is added to the engine.handles, and this gives the error you see in the test failure "'list' object has no attribute 'close'"

[lababidi]

@jreback @jorisvandenbossche Thank you for the insight. The handles is new to me. I must be passing the wrong thing through. I'll have to understand the logic correctly on what we're aiming for.

[dhimmel]

What is the remaining work that needs to be done for this pull request? I'm willing to give it a try, but need a little guidance to get started.

[jorisvandenbossche]

@dhimmel To start with, you can look into the failing tests. As that is of course the minimum for getting this merged.
The failure has to do with passing a list of lines to TextParser, which now no longer works.