BUG: Ensure min_itemsize is always a list (#11412) #14728

toobaz · 2016-11-24T11:39:41Z

closes min_itemsize not working on MultiIndex columns for Series, with format="table" #11412
passes git diff upstream/master | flake8 --diff
tests added / passed
whatsnew entry

(I plan to fix #10381 and #12154 next, but first we need coherence on type(data_columns) - and Index seems to me the best option.)

codecov-io · 2016-11-24T12:22:18Z

Current coverage is 85.21% (diff: 50.00%)

Merging #14728 into master will decrease coverage by 0.05%

@@             master     #14728   diff @@
==========================================
  Files           144        143     -1   
  Lines         50947      50804   -143   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43445      43293   -152   
- Misses         7502       7511     +9   
  Partials          0          0

Powered by Codecov. Last update 5d0e157...1f70a6e

jreback · 2016-11-24T15:34:37Z

this is technically an API change because we currently store the data_columns as a pickled list (there is a separate issues on actually storing this as a table). I am surpised this didn't break.

So i would prefer to keep this as a list for now, though see if you can create a tests which fails on this.

jreback · 2016-11-24T15:35:27Z

the issue you reference is closed. so what issue is this fixing?

toobaz · 2016-11-24T15:59:29Z

Regarding the issue: yes, it is closed, but not solved, it is different from the #11364 you referred to. I can open a new issue if you prefer. But the thing is: I am not solving #11364 here, I am solving #11412, which is different.

Regarding the data_columns storage: I had missed that indeed, can you give me a pointer to understand more (I don't see that use of pickle in pytables.py)?

My motivation was that in some cases data_columns already is an index (e.g. at line 4257 of pytables.py), but then I can certainly fix that (as I was doing in the previous #12252).

jreback · 2016-11-24T16:12:25Z

then are you solving this? #10381

jreback · 2016-11-24T16:14:58Z

https://github.com/pandas-dev/pandas/blob/master/pandas/io/pytables.py#L3129

you would need to be pretty careful about this (though not 100% sure it actually matters). It would be nice to be consistent, though everything is a list. yes Index has the same semantics. but I don't want to change this unless it stays consistent (which is already a list for almost everything else).

so bottom line is, keep it a list. If you have a repro where its NOT a list. then pls show that.

toobaz · 2016-11-24T19:14:01Z

OK, new version works with lists rather than indices.

No, I'm not fixing #10381, otherwise I wouldn't have written that I plan to fix it when I opened the PR ...

toobaz · 2016-11-25T16:39:31Z

The AppVeyor failure is a bug in AppVeyor, right?

jreback · 2016-11-25T16:54:48Z

I restarted. it sometimes does get stuck.

toobaz · 2016-11-27T16:24:41Z

again...

toobaz · 2016-12-03T22:00:36Z

ping

jreback · 2016-12-04T17:59:12Z

doc/source/whatsnew/v0.19.2.txt

+
+
+
+- Bug in ``HDFStore.append()`` with Series and 'index' appearing in ``min_itemsize`` (:issue:`11412`)


a little bit more verbose as I don't understand what you are fixing.

jreback · 2016-12-04T17:59:52Z

pandas/io/pytables.py

@@ -4254,7 +4254,7 @@ def write(self, obj, data_columns=None, **kwargs):
        if data_columns is None:
            data_columns = []
        elif data_columns is True:
-            data_columns = obj.columns[:]
+            data_columns = list(obj.columns)


was just fixed in #14791

jreback · 2016-12-04T18:00:04Z

pandas/io/pytables.py

@@ -4153,7 +4153,7 @@ def write(self, obj, data_columns=None, **kwargs):
            obj = DataFrame({name: obj}, index=obj.index)
            obj.columns = [name]
        return super(AppendableSeriesTable, self).write(
-            obj=obj, data_columns=obj.columns, **kwargs)
+            obj=obj, data_columns=list(obj.columns), **kwargs)


use obj.columns.tolist() to be consistent

jreback · 2016-12-04T18:01:24Z

doc/source/whatsnew/v0.19.2.txt

+
+
+
+


put underneath the whatsnew for #14791

Closes pandas-dev#11412

toobaz · 2016-12-05T15:16:06Z

Stuck again

jreback · 2016-12-05T23:44:35Z

thanks!

closes #11412 Author: Pietro Battiston <[email protected]> Closes #14728 from toobaz/minitemsizefix and squashes the following commits: e25cd1f [Pietro Battiston] Whatsnew b9bb88f [Pietro Battiston] Tests for previous commit 6406ee8 [Pietro Battiston] BUG: Ensure min_itemsize is always a list (cherry picked from commit 53bf1b2)

toobaz mentioned this pull request Nov 24, 2016

BUG: Ensure data_columns is always a list (i.e. min_itemsize can exte… #12252

Closed

jreback added the IO HDF5 read_hdf, HDFStore label Nov 24, 2016

toobaz force-pushed the minitemsizefix branch from e3fb504 to 1f70a6e Compare November 24, 2016 19:12

jorisvandenbossche changed the title ~~Minitemsizefix~~ BUG: Ensure min_itemsize is always a list (#11412) Nov 25, 2016

jreback reviewed Dec 4, 2016

View reviewed changes

doc/source/whatsnew/v0.19.2.txt

Copy link

Contributor

jreback Dec 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put underneath the whatsnew for #14791

toobaz added 3 commits December 5, 2016 08:48

BUG: Ensure min_itemsize is always a list

6406ee8

Closes pandas-dev#11412

Tests for previous commit

b9bb88f

Whatsnew

e25cd1f

toobaz force-pushed the minitemsizefix branch from 1f70a6e to e25cd1f Compare December 5, 2016 07:49

jreback added this to the 0.19.2 milestone Dec 5, 2016

jreback added the Bug label Dec 5, 2016

jreback closed this in 53bf1b2 Dec 5, 2016

toobaz deleted the minitemsizefix branch December 6, 2016 07:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Ensure min_itemsize is always a list (#11412) #14728

BUG: Ensure min_itemsize is always a list (#11412) #14728

toobaz commented Nov 24, 2016 •

edited by jorisvandenbossche

Loading

codecov-io commented Nov 24, 2016 •

edited

Loading

jreback commented Nov 24, 2016

jreback commented Nov 24, 2016

toobaz commented Nov 24, 2016

jreback commented Nov 24, 2016

jreback commented Nov 24, 2016

toobaz commented Nov 24, 2016

toobaz commented Nov 25, 2016

jreback commented Nov 25, 2016

toobaz commented Nov 27, 2016

toobaz commented Dec 3, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

toobaz commented Dec 5, 2016

jreback commented Dec 5, 2016




		- Bug in ``HDFStore.append()`` with Series and 'index' appearing in ``min_itemsize`` (:issue:`11412`)

BUG: Ensure min_itemsize is always a list (#11412) #14728

BUG: Ensure min_itemsize is always a list (#11412) #14728

Conversation

toobaz commented Nov 24, 2016 • edited by jorisvandenbossche Loading

codecov-io commented Nov 24, 2016 • edited Loading

Current coverage is 85.21% (diff: 50.00%)

jreback commented Nov 24, 2016

jreback commented Nov 24, 2016

toobaz commented Nov 24, 2016

jreback commented Nov 24, 2016

jreback commented Nov 24, 2016

toobaz commented Nov 24, 2016

toobaz commented Nov 25, 2016

jreback commented Nov 25, 2016

toobaz commented Nov 27, 2016

toobaz commented Dec 3, 2016

jreback Dec 4, 2016

Choose a reason for hiding this comment

jreback Dec 4, 2016

Choose a reason for hiding this comment

jreback Dec 4, 2016

Choose a reason for hiding this comment

jreback Dec 4, 2016

Choose a reason for hiding this comment

toobaz commented Dec 5, 2016

jreback commented Dec 5, 2016

toobaz commented Nov 24, 2016 •

edited by jorisvandenbossche

Loading

codecov-io commented Nov 24, 2016 •

edited

Loading