API/BUG: PandasArray __setitem__ can change underlying buffer

[jbrockmendel]

Discussed briefly in #28119, the ndarray backing a PandasArray can be swapped out by setitem, which can have surprising behavior for views

arr = pd.array([1, 2, 3])
view1 = arr.view()
view2 = arr[:]
view3 = np.asarray(arr)

arr[0] = 9
assert view1[0] == 9
assert view2[0] == 9 
assert view3[0] == 9

arr[1] = 2.5
view1[-1] = 5
assert arr[-1] == 5  # FAIL

[jbrockmendel]

@TomAugspurger sure this is closed by #28119? My understanding was that specifically punted on this topic.

[TomAugspurger]

Probably OK to keep open.

…

On Tue, Sep 17, 2019 at 4:41 PM jbrockmendel ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> sure this is closed by #28119 <#28119>? My understanding was that specifically punted on this topic. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28150?email_source=notifications&email_token=AAKAOIRDQLZUEDQFH26IBFLQKFFKBA5CNFSM4IPQMHGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD657YUA#issuecomment-532413520>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIUJD2JJ73P2LELVH73QKFFKBANCNFSM4IPQMHGA> .

[TomAugspurger]

So the way NumPy handles this is by not allowing __setitem__ to change the dtype of the array.

In [18]: a = np.array([1, 2])

In [19]: a[0] = 1.5

In [20]: a
Out[20]: array([1, 2])

On master, we now match that behavior. The original code snippet passes.

I think the remaining question is: do we want this behavior? Silently truncating the input? Or would we rather raise when the input cannot be faithfully inserted into an array with that dtype (can we even determine that)?

[jreback]

maybe we should start warning on this? I think we are going to eventually be able to 'fix' this, but would be a biggish change.

[jbrockmendel]

maybe we should start warning on this? I think we are going to eventually be able to 'fix' this, but would be a biggish change.

Is anyone actually using PandasArray? I'm hoping they'll eventually back Block, but until then, it isn't clear to me that this is really user-facing.

I think the remaining question is: do we want this behavior? Silently truncating the input? Or would we rather raise when the input cannot be faithfully inserted into an array with that dtype (can we even determine that)?

Truncating is basically the same as ndarray behavior, which I'm fine with. Raising is basically the same as DTA/TDA/PA/Categorical (more general EA is less well-defined AFAICT) behavior, which I'm also fine with. If we make it raise, I think that means we can get rid of Block._can_hold_element (although we would be replacing it with a try/except, which I'm wary of).

[TomAugspurger]

Is anyone actually using PandasArray? I'm hoping they'll eventually back Block, but until then, it isn't clear to me that this is really user-facing.

The main way would be via Series[...].array for an ndarray-backed Series. I think we can afford to change this without a warning.

---

Are we likely to do this for 1.0?

[TomAugspurger]

I don't think this is a blocker for 1.0.

[jbrockmendel]

closed by #30523.