ENH: feather support in the pandas IO api #14383

jreback · 2016-10-10T00:34:48Z

jreback · 2016-10-10T00:35:46Z

pandas/io/feather_format.py

+
+    # validate that we do not have any non-string object dtypes
+    # as these 'work', but will not properly de-serialize
+    valid_types = set(['string', 'unicode'])


this provides a check for this serialization issue: wesm/feather#240

{'string', 'unicode'} works too FYI

fixed....old habits :>

jreback · 2016-10-10T00:36:13Z

cc @wesm

codecov-io · 2016-10-10T12:56:28Z

Current coverage is 84.78% (diff: 82.35%)

Merging #14383 into master will increase coverage by 0.12%

@@             master     #14383   diff @@
==========================================
  Files           144        145     +1   
  Lines         51056      51090    +34   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43225      43316    +91   
+ Misses         7831       7774    -57   
  Partials          0          0

Powered by Codecov. Last update 72786cc...aa3c4b4

wesm · 2016-10-10T15:41:15Z

pandas/io/feather_format.py

+
+    # must be a Index
+    if isinstance(df.columns, MultiIndex):
+        raise ValueError("feather does not serializing a MultiIndex "


wesm · 2016-10-10T15:41:36Z

pandas/io/feather_format.py

+
+    # validate that we do not have any non-string object dtypes
+    # as these 'work', but will not properly de-serialize
+    objects = df.select_dtypes(include=['object'])


does this copy?

yes, fixed another way

wesm · 2016-10-10T15:42:37Z

pandas/io/feather_format.py

+        msg = ("The following columns are not supported to serialize "
+               "to the feather-format:\n\n"
+               "{}".format(invalid.to_string()))
+        raise ValueError(msg)


I think this whole function should be pushed down into Feather itself, and the dtype inference can be handled there too

Same goes for the unit tests...

jorisvandenbossche · 2016-10-22T10:00:46Z

Shall we target this for 0.20?

jreback · 2016-10-24T10:29:13Z

some of these checks can be obviated by: wesm/feather#244 & wesm/feather#243

ERR: nicer error message on passing duplicate columns, xref #53 ERR: nice error message on serializing python objects, closes #240 should be after #239 xref pandas-dev/pandas#14383 Author: Jeff Reback <[email protected]> Closes #244 from jreback/infer_dtype and squashes the following commits: 091ee60 [Jeff Reback] fix pandas < 0.19.0 compat 6ef3a47 [Jeff Reback] fixed up mixed with embedded nulls tests 560b9e6 [Jeff Reback] ERR: nicer error message on passing duplicate columns, xref #53

jreback · 2016-12-16T11:31:57Z

this is ready to go.

@jorisvandenbossche
@wesm

TomAugspurger · 2016-12-16T13:35:43Z

pandas/io/feather_format.py

+    # validate that we have only a default index
+    # raise on anything else as we don't serialize the index
+
+    if not isinstance(df.index, (RangeIndex, Int64Index)):


Can this be simplified to isinstance(df.index, Int64Index)?

In [6]: idx Out[6]: RangeIndex(start=0, stop=10, step=1) In [7]: isinstance(idx, pd.Int64Index) Out[7]: True

Or is that not by design?

I wonder if you should also check whether df.index.name is not None, and raise if it's set (or maybe warn?). I'm thinking about this too a bit in the JSON Table Schema stuff.

hmm, yeah I can change the instance.

hmm, yes I will test for meta-data as well.

jorisvandenbossche · 2016-12-16T13:33:49Z

doc/source/io.rst

+.. ipython:: python
+
+   pd.read_feather('example.fth')
+


Maybe also show the dtypes? (so you see it is preserverd automatically)

jorisvandenbossche · 2016-12-16T13:34:40Z

doc/source/api.rst

+.. autosummary::
+   :toctree: generated/
+
+   read_feather


also to_feather in the dataframe section?

jorisvandenbossche · 2016-12-16T13:37:47Z

doc/source/io.rst

+- This is a newer library, and the format, though stable, is not guaranteed to be backward compatible
+  to the earlier versions.
+- The format will NOT write an ``Index``, or ``MultiIndex`` for the ``DataFrame`` and will raise an
+  error if a non-default one is provided. You can simply ``.reset_index()`` in order to store the index.


Additional point: Non-string column names ?

and duplicate column names

yep, these are raised automaticaly by feather now (as of 3.1)

jorisvandenbossche · 2016-12-16T13:38:37Z

pandas/io/feather_format.py

@@ -0,0 +1,93 @@
+""" feather-format compat """


Very minor, but can we call this file just feather.py ?

ah, missed that the package is imported like that, confused by the feather-format package name

yeah that is an annoying 'feature' in python!

jorisvandenbossche · 2016-12-16T13:40:10Z

pandas/io/feather_format.py

+        # give a nice error message
+        raise ImportError("the feather-format library is not installed\n"
+                          "you can install via conda\n"
+                          "conda install feather-format -c conda-forge")


hmm, on linux I suppose this is ok (maybe mac), but certainly not on windows.

@wesm ?

It'll work on Windows if you have the right compiler setup. Can recommend both here I suppose

jorisvandenbossche · 2016-12-16T13:41:49Z

pandas/io/feather_format.py

+    # raise on anything else as we don't serialize the index
+
+    if not isinstance(df.index, (RangeIndex, Int64Index)):
+        raise ValueError("feather does not serializing {} "


either "does not support serializing" or "does not serialize"

jorisvandenbossche · 2016-12-16T13:42:24Z

pandas/io/feather_format.py

+                             type(df.index)))
+
+    if not df.index.equals(RangeIndex.from_range(range(len(df)))):
+        raise ValueError("feather does not serializing a non-default index "


the same here

jreback · 2016-12-16T13:55:25Z

all comments addressed (except if we want to recommend pip installs)

jreback · 2016-12-26T21:26:36Z

any final comments. @wesm ?

going to merge in the next day or 2.

wesm

LGTM

wesm · 2016-12-26T22:28:54Z

pandas/io/feather_format.py

+        # give a nice error message
+        raise ImportError("the feather-format library is not installed\n"
+                          "you can install via conda\n"
+                          "conda install feather-format -c conda-forge")


It'll work on Windows if you have the right compiler setup. Can recommend both here I suppose

closes pandas-dev#13092

jreback · 2016-12-26T22:46:48Z

ok added pip install instructions as well.

jorisvandenbossche · 2016-12-27T16:24:06Z

@jreback we should also add feather-format to the list of the doc building requirements (ci/requirements-2.7_DOC_BUILD.run)

jreback · 2016-12-27T17:44:05Z

yep will do

jreback · 2016-12-28T11:25:32Z

@jorisvandenbossche 3e3434b

FYI, has to be in the .sh script because we don't have conda-forge in the default channel list on travis; we get some weird override behavior with some packages, I think because of some incorrect meta-data, but these are really old and not likely to be changed. The .sh solves this problem by simply allowing arbitrary stuff for any build which needs it.

jorisvandenbossche · 2016-12-29T00:28:36Z

@jreback Seems to be working now! There was only still a small typo: 975a440

jorisvandenbossche · 2016-12-29T00:34:12Z

@jreback @wesm A more stylistic question: Jeff now used .fth as the file extension, but I think I have mainly seen .feather (in blogposts about it). Best to be consistent here. @wesm do you prefer one of both?

wesm · 2016-12-29T01:25:21Z

We've been using .feather, maybe stick with that.

jorisvandenbossche · 2016-12-29T12:06:01Z

OK, done: e27b296

jreback added Enhancement IO Data IO issues that don't fit into a more specific label labels Oct 10, 2016

jreback added this to the 0.19.1 milestone Oct 10, 2016

jreback commented Oct 10, 2016

View reviewed changes

jreback force-pushed the feather branch from 6c9d43a to 55af5cd Compare October 10, 2016 12:56

jreback force-pushed the feather branch 3 times, most recently from 306987a to c0d9558 Compare October 10, 2016 15:24

wesm reviewed Oct 10, 2016

View reviewed changes

jreback force-pushed the feather branch from c0d9558 to 9774163 Compare October 11, 2016 00:20

jreback mentioned this pull request Oct 11, 2016

ERR: [Python] infer_dtypes for better error messages on write_dataframe wesm/feather#244

Closed

jreback modified the milestones: 0.20.0, 0.19.1 Oct 24, 2016

jreback force-pushed the feather branch from 9774163 to dfb45e4 Compare October 24, 2016 10:28

jreback force-pushed the feather branch 2 times, most recently from 27eb06a to 8b31e54 Compare October 26, 2016 10:31

jreback force-pushed the feather branch 4 times, most recently from de574d5 to 48f64c1 Compare November 30, 2016 23:42

jreback force-pushed the feather branch 2 times, most recently from c1a4012 to 182aa28 Compare December 13, 2016 11:39

jreback force-pushed the feather branch 3 times, most recently from b3945c4 to 0123d1c Compare December 16, 2016 11:02

jreback force-pushed the feather branch from 0123d1c to 71c4007 Compare December 16, 2016 11:31

jreback force-pushed the feather branch from 71c4007 to 73b2fe6 Compare December 16, 2016 11:33

TomAugspurger reviewed Dec 16, 2016

View reviewed changes

jorisvandenbossche reviewed Dec 16, 2016

View reviewed changes

jreback force-pushed the feather branch 2 times, most recently from 0c952d8 to 11a0e30 Compare December 16, 2016 13:55

jreback force-pushed the feather branch 6 times, most recently from ae9e4c5 to 4bad608 Compare December 21, 2016 00:06

jreback force-pushed the feather branch from 4bad608 to aa3c4b4 Compare December 26, 2016 17:31

wesm approved these changes Dec 26, 2016

View reviewed changes

ENH: feather support in the pandas IO api

3ede160

closes pandas-dev#13092

jreback force-pushed the feather branch from aa3c4b4 to 3ede160 Compare December 26, 2016 22:46

jreback closed this in eecfa88 Dec 26, 2016

ENH: feather support in the pandas IO api #14383

ENH: feather support in the pandas IO api #14383

Conversation

jreback commented Oct 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Oct 10, 2016

codecov-io commented Oct 10, 2016 • edited Loading

Current coverage is 84.78% (diff: 82.35%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Oct 22, 2016

jreback commented Oct 24, 2016

jreback commented Dec 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 16, 2016 • edited Loading

jreback commented Dec 26, 2016

wesm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 26, 2016

jorisvandenbossche commented Dec 27, 2016

jreback commented Dec 27, 2016

jreback commented Dec 28, 2016 • edited Loading

jorisvandenbossche commented Dec 29, 2016

jorisvandenbossche commented Dec 29, 2016

wesm commented Dec 29, 2016

jorisvandenbossche commented Dec 29, 2016

codecov-io commented Oct 10, 2016 •

edited

Loading

jreback commented Dec 16, 2016 •

edited

Loading

jreback commented Dec 28, 2016 •

edited

Loading