Skip to content

REF: unstack #33474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 12, 2020
Merged

REF: unstack #33474

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7c33dc2
REF: move shared calls up the stack
jbrockmendel Apr 9, 2020
bc19cad
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Apr 9, 2020
45c7a8f
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Apr 9, 2020
b2feffe
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Apr 10, 2020
80e0204
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Apr 11, 2020
8f0e5d4
Comments
jbrockmendel Apr 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 6 additions & 9 deletions pandas/core/internals/blocks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1384,15 +1384,13 @@ def equals(self, other) -> bool:
return False
return array_equivalent(self.values, other.values)

def _unstack(self, unstacker, new_columns, fill_value, value_columns):
def _unstack(self, unstacker, fill_value, new_placement):
"""
Return a list of unstacked blocks of self

Parameters
----------
unstacker : reshape._Unstacker
new_columns : Index
All columns of the unstacked BlockManager.
fill_value : int
Only used in ExtensionBlock._unstack

Expand All @@ -1403,17 +1401,17 @@ def _unstack(self, unstacker, new_columns, fill_value, value_columns):
mask : array_like of bool
The mask of columns of `blocks` we should keep.
"""
new_items = unstacker.get_new_columns(value_columns)
new_placement = new_columns.get_indexer(new_items)
new_values, mask = unstacker.get_new_values(
self.values.T, fill_value=fill_value
)

mask = mask.any(0)
# TODO: in all tests we have mask.all(); can we rely on that?

new_values = new_values.T[mask]
new_placement = new_placement[mask]

blocks = [make_block(new_values, placement=new_placement)]
blocks = [self.make_block_same_class(new_values, placement=new_placement)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jbrockmendel I think it's just this line that caused the regression in #37115 and that we don't need to revert the complete PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what kind of block(s) do you end up with instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

corrupted? an IntBlock with dtype float64

>>> pd.__version__
'1.2.0.dev0+1432.g5fdf642368'
>>>
>>> df1 = pd.DataFrame(
...     {
...         "a": ["A", "A", "B"],
...         "b": ["ca", "cb", "cb"],
...         "v": [10] * 3,
...     }
... )
>>> df1 = df1.set_index(["a", "b"])
>>> df1["is_"] = 1
>>> df1
       v  is_
a b
A ca  10    1
  cb  10    1
B cb  10    1
>>>
>>> df1._data
BlockManager
Items: Index(['v', 'is_'], dtype='object')
Axis 1: MultiIndex([('A', 'ca'),
            ('A', 'cb'),
            ('B', 'cb')],
           names=['a', 'b'])
IntBlock: slice(0, 1, 1), 1 x 3, dtype: int64
IntBlock: slice(1, 2, 1), 1 x 3, dtype: int64
>>>
>>> df2 = df1.unstack("b")
>>> df2
      v        is_
b    ca    cb   ca   cb
a
A  10.0  10.0  1.0  1.0
B   NaN  10.0  NaN  1.0
>>>
>>> df2._data
BlockManager
Items: MultiIndex([(  'v', 'ca'),
            (  'v', 'cb'),
            ('is_', 'ca'),
            ('is_', 'cb')],
           names=[None, 'b'])
Axis 1: Index(['A', 'B'], dtype='object', name='a')
IntBlock: slice(0, 2, 1), 2 x 2, dtype: float64
IntBlock: slice(2, 4, 1), 2 x 2, dtype: float64
>>>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run the test suite with just this change reverted and all tests pass. will put up a PR shortly.

return blocks, mask

def quantile(self, qs, interpolation="linear", axis: int = 0):
Expand Down Expand Up @@ -1878,7 +1876,7 @@ def where(

return [self.make_block_same_class(result, placement=self.mgr_locs)]

def _unstack(self, unstacker, new_columns, fill_value, value_columns):
def _unstack(self, unstacker, fill_value, new_placement):
# ExtensionArray-safe unstack.
# We override ObjectBlock._unstack, which unstacks directly on the
# values of the array. For EA-backed blocks, this would require
Expand All @@ -1888,10 +1886,9 @@ def _unstack(self, unstacker, new_columns, fill_value, value_columns):
n_rows = self.shape[-1]
dummy_arr = np.arange(n_rows)

new_items = unstacker.get_new_columns(value_columns)
new_placement = new_columns.get_indexer(new_items)
new_values, mask = unstacker.get_new_values(dummy_arr, fill_value=-1)
mask = mask.any(0)
# TODO: in all tests we have mask.all(); can we rely on that?

blocks = [
self.make_block_same_class(
Expand Down
5 changes: 4 additions & 1 deletion pandas/core/internals/managers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1461,8 +1461,11 @@ def unstack(self, unstacker, fill_value) -> "BlockManager":

for blk in self.blocks:
blk_cols = self.items[blk.mgr_locs.indexer]
new_items = unstacker.get_new_columns(blk_cols)
new_placement = new_columns.get_indexer(new_items)

blocks, mask = blk._unstack(
unstacker, new_columns, fill_value, value_columns=blk_cols,
unstacker, fill_value, new_placement=new_placement
)

new_blocks.extend(blocks)
Expand Down
5 changes: 4 additions & 1 deletion pandas/core/reshape/reshape.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ def sorted_labels(self):
indexer, to_sort = self._indexer_and_to_sort
return [l.take(indexer) for l in to_sort]

def _make_sorted_values(self, values):
def _make_sorted_values(self, values: np.ndarray) -> np.ndarray:
indexer, _ = self._indexer_and_to_sort

sorted_values = algos.take_nd(values, indexer, axis=0)
Expand Down Expand Up @@ -205,6 +205,9 @@ def get_new_values(self, values, fill_value=None):

# we can simply reshape if we don't have a mask
if mask_all and len(values):
# TODO: Under what circumstances can we rely on sorted_values
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if the we are sorted already then you can simplify, but I am not sure we can make that guarantee.

# matching values? When that holds, we can slice instead
# of take (in particular for EAs)
new_values = (
sorted_values.reshape(length, width, stride)
.swapaxes(1, 2)
Expand Down