Add ZIP file decompression and TestCompression. #12175

lababidi · 2016-01-29T15:15:25Z

jreback · 2016-01-29T15:23:23Z

pandas/io/parsers.py

                klass = FixedWidthFieldParser
+            else:  #default to engine == 'python':


why did you modify this?

I modified this to default to the Python engine. If this is not the wanted functionality. I can remove it.

just change code that is relevant.

will do. Does this need to a separate PR?

not sure why you are changing this in the first place.

For the use case that engine is neither 'python' nor 'python-fwf'.

Is it possible for this to happen?

no, that would be an exception and should raise a ValueError (do this in another issue/PR)

ok, thanks for the clarification.

lababidi · 2016-01-29T16:40:17Z

@jreback The idea of the Mixin makes sense. Took me a bit to wrap my head around it. I had to add self.engine to a couple of the tests because bzip decompression will not raise an exception and so the Compression Mixin needs to check what engine it is using to make sure the test runs correctly.

lababidi · 2016-01-29T18:57:39Z

@jreback I think this is ready for your review.

max-sixty · 2016-01-29T19:01:01Z

pandas/io/common.py

+                f = zip_file.open(file_name)
+            else:
+                raise ValueError('ZIP file contains multiple files {}',
+                                 zip_file.filename)


You need a .format here

max-sixty · 2016-01-29T19:03:39Z

Jeff is going to ask you to squash your commits into one, as per the contributing docs.
Nice job overall!

TomAugspurger · 2016-01-29T19:10:21Z

@MaximilianR no need for squashing anymore thanks to https://github.com/pydata/pandas/blob/master/scripts/merge-py.py ;)

EDIT: which reminds me, CONTRIBUTING.md needs to be updated. Will do this weekend unless someone beats me to it.

lababidi · 2016-01-29T19:18:40Z

@MaximilianR @TomAugspurger Thanks for the comments guys!

max-sixty · 2016-01-29T20:24:38Z

@TomAugspurger Cool!

jreback · 2016-01-29T21:25:44Z

pls squash

even though I can do it - it's much cleaner from a future reader perspective on a smaller change

lababidi · 2016-01-29T21:43:19Z

@jreback Squashed. Thanks.

jreback · 2016-01-30T15:02:53Z

pandas/io/tests/test_parsers.py

+            result = self.read_csv(path, compression='zip')
+            tm.assert_frame_equal(result, expected)
+
+            result = self.read_csv(open(path, 'rb'), compression='zip')


do this in a with block to make sure the file is closed

jreback · 2016-01-30T15:06:07Z

looks pretty good. just some minor stylistic comments.

jreback · 2016-01-30T15:07:35Z

======================================================================
ERROR: test_to_csv_compression_value_error (pandas.tests.frame.test_to_csv.TestDataFrameToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/frame/test_to_csv.py", line 998, in test_to_csv_compression_value_error
    filename, compression="zip")
  File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 1952, in assertRaises
    _callable(*args, **kwargs)
  File "/home/travis/build/pydata/pandas/pandas/core/frame.py", line 1338, in to_csv
    formatter.save()
  File "/home/travis/build/pydata/pandas/pandas/core/format.py", line 1524, in save
    compression=self.compression)
  File "/home/travis/build/pydata/pandas/pandas/io/common.py", line 346, in _get_handle
    zip_file = zipfile.ZipFile(path)
  File "/home/travis/miniconda/envs/pandas/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/home/travis/miniconda/envs/pandas/lib/python2.7/zipfile.py", line 811, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file

put a test in for this error as well (e.g. try to open a non-zipfile); I don't think you need any code changes though, you can just let it raise.

lababidi · 2016-02-01T21:04:36Z

@jreback Good ideas. I think I covered all your requests.

jreback · 2016-02-01T22:38:02Z

@lababidi ok, this lgtm.

just need a whatsnew note (you can put kind of what you did in the addition to the doc-string). pls add, ping when green.

jreback · 2016-03-18T20:26:52Z

pandas/io/tests/test_parsers.py

+
+            file_name = 'test_file.zip'
+            with tm.ensure_clean(file_name) as path:
+                tmp = zipfile.ZipFile(path, mode='w')


you didn't change anything here

@jreback what line are you referring to? zipfile.ZipFile?

no, your context managers are nested, but don't need to be

with open(self.csv1, 'rb') as data_file: data = data_file.read() data can be used here

@jreback thanks for clarifying. I was using the previous convention. I'll clean this up now.

jreback · 2016-03-18T21:31:49Z

pandas/io/parsers.py

+compression : {'gzip', 'bz2', 'zip', 'infer', None}, default 'infer'
+    For on-the-fly decompression of on-disk data. If 'infer', then use gzip,
+    bz2 or zip if filepath_or_buffer is a string ending in '.gz', '.bz2' or
+    '.zip', respectively, and no decompression otherwise. New in 0.18.0: ZIP


so add in a versionadded tag and put the 0.18.0 stuff there

lababidi · 2016-03-18T22:01:36Z

@jreback previous tests passed. The most recent push only changed the version in the docstring
https://travis-ci.org/pydata/pandas/builds/117001384

jreback · 2016-03-18T22:04:45Z

pandas/io/parsers.py

+compression : {'gzip', 'bz2', 'zip', 'infer', None}, default 'infer'
+    For on-the-fly decompression of on-disk data. If 'infer', then use gzip,
+    bz2 or zip if filepath_or_buffer is a string ending in '.gz', '.bz2' or
+    '.zip', respectively, and no decompression otherwise. New in 0.18.1: ZIP


need a versionadded tag here

jreback · 2016-03-18T22:11:16Z

@lababidi ok looks pretty good. only changes are to use assertRaisesRegexp when you assert for zip in order to make sure the correct messages are raised (as there are multiple possibilites).

pls make that and ping when green.

lababidi · 2016-03-21T21:23:34Z

@jreback could you help me? the test only failed on the following:

--------------------------------------------------------------------------------------------------------------
#176 nose.failure.Failure.runTest: direct creation of extension dtype datetime64[ns, UTC] is not supported ATM

jreback · 2016-03-21T21:26:19Z

not sure where you are seeing this. your 2.7 tests is failing because of a linting issue (line too long)

git diff master | flake8 --diff

lababidi · 2016-03-21T21:38:28Z

@jreback it's in the Travis results

jreback · 2016-03-21T21:54:01Z

I restarted that job, though you need to repush anyhow (lint error). Never saw that one before; I think its a crash in something else, so let's see if it recurs.

jreback · 2016-03-22T13:18:59Z

@lababidi ok this passed, except for the lint check. pls fix and repush, ping when green.

Fix PEP8 issues. Change Compression to be a Mixin. Add Compression Mixin correctly with current Tests. Add .format, Rename Compression, with-block, empty zip, bad-zip

lababidi · 2016-03-22T16:11:01Z

Thank you @jreback for your help and patience with this. I'll help out on the other issues soon.

jreback · 2016-03-22T16:16:56Z

@lababidi no thank you!

jreback added IO Data IO issues that don't fit into a more specific label IO CSV read_csv, to_csv labels Jan 29, 2016

jreback reviewed Jan 29, 2016
View reviewed changes

max-sixty reviewed Jan 29, 2016
View reviewed changes

lababidi force-pushed the feature_zip branch from c8f4dda to 8c7bee5 Compare January 29, 2016 21:42

jreback reviewed Jan 30, 2016
View reviewed changes

lababidi force-pushed the feature_zip branch from 1302942 to 539e9b7 Compare February 1, 2016 21:06

jreback added this to the 0.18.0 milestone Feb 1, 2016

lababidi force-pushed the feature_zip branch 2 times, most recently from ccfc4a7 to 032362c Compare February 2, 2016 15:39

lababidi force-pushed the feature_zip branch from 1919273 to 1f65cd4 Compare March 18, 2016 20:22

jreback reviewed Mar 18, 2016
View reviewed changes

lababidi force-pushed the feature_zip branch from 1f65cd4 to e6ae96c Compare March 18, 2016 20:31

jreback mentioned this pull request Mar 18, 2016

ENH: xz compression in to_csv() resolves #11852 #12668

Closed

4 tasks

lababidi force-pushed the feature_zip branch from e6ae96c to dd9fca0 Compare March 18, 2016 21:06

jreback reviewed Mar 18, 2016
View reviewed changes

lababidi force-pushed the feature_zip branch from dd9fca0 to 725dd2c Compare March 18, 2016 22:00

jreback reviewed Mar 18, 2016
View reviewed changes

lababidi force-pushed the feature_zip branch 2 times, most recently from f57a7ee to daae2aa Compare March 21, 2016 18:33

Add ZIP file decompression and TestCompression.

68eb884

Fix PEP8 issues. Change Compression to be a Mixin. Add Compression Mixin correctly with current Tests. Add .format, Rename Compression, with-block, empty zip, bad-zip

lababidi force-pushed the feature_zip branch from daae2aa to 68eb884 Compare March 22, 2016 13:46

jreback closed this in 7ee6d08 Mar 22, 2016

jreback mentioned this pull request Mar 22, 2016

CLN: consolidate compression inference as much as possible #12688

Closed

		klass = FixedWidthFieldParser
		else: #default to engine == 'python':

Add ZIP file decompression and TestCompression. #12175

Add ZIP file decompression and TestCompression. #12175

Conversation

lababidi commented Jan 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lababidi commented Jan 29, 2016

lababidi commented Jan 29, 2016

Choose a reason for hiding this comment

max-sixty commented Jan 29, 2016

TomAugspurger commented Jan 29, 2016

lababidi commented Jan 29, 2016

max-sixty commented Jan 29, 2016

jreback commented Jan 29, 2016

lababidi commented Jan 29, 2016

Choose a reason for hiding this comment

jreback commented Jan 30, 2016

jreback commented Jan 30, 2016

lababidi commented Feb 1, 2016

jreback commented Feb 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lababidi commented Mar 18, 2016

Choose a reason for hiding this comment

jreback commented Mar 18, 2016

lababidi commented Mar 21, 2016

jreback commented Mar 21, 2016

lababidi commented Mar 21, 2016

jreback commented Mar 21, 2016

jreback commented Mar 22, 2016

lababidi commented Mar 22, 2016

jreback commented Mar 22, 2016