COMPAT: Utilize copy keyword in __array__

[mroeschke]

#57172 add a copy keyword that will be passed in with __array__ in Numpy 2.0. Currently it is unused but it would be useful to respect that option when passed

[jorisvandenbossche]

Linking the numpy issue I opened to discuss the expected semantics for this: numpy/numpy#25941 (in several places we now also ignore the dtype keyword, but for copy that's something we shouldn't do)

[rhshadrach]

I believe this came up again in a new issue. Should this be a blocker for 3.0?

[mroeschke]

IMO I don't think this "bug" should block a release, but would be very nice to fix for 3.0

[mg3146]

Hi - this is caused fairly large blocking issues in a production environment for us. If there was a way to roll a fix out , that would be great...

[seberg]

This is a very simple thing to hot-fix @mg3146 you could probably do a PR yourself even. I agree it is a relatively large issue (because it can lead to silently modified dataframes on existing code).
Making a defensive copy may not be always ideal, but is totally fine. Only copy is False might be tricky, but copy=False is a new feature, so hopefully not a big deal in practice.

If you can't (or do not want) guess copy=False correctly, I think it is fine to just reject it always. That might be a small argument for fixing it in 3.0: avoid breaking code that thinks copy=False work when you turn it into an error.

[bashtage]

This bug is affecting statsmodels. Internally we mostly use ndarrays for predictability, and then wrap results as needed for pandas inputs. To do this, we often call np.array(list | DataFrame | Series | nparray, copy=True) when it is efficient that we own a copy of the data, e.g, when needing to sort. Since pandas does not return a copy, we end up modifying the original data, which is highly undesirable.

[mg3146]

This is a very simple thing to hot-fix @mg3146 you could probably do a PR yourself even. I agree it is a relatively large issue (because it can lead to silently modified dataframes on existing code). Making a defensive copy may not be always ideal, but is totally fine. Only copy is False might be tricky, but copy=False is a new feature, so hopefully not a big deal in practice.

If you can't (or do not want) guess copy=False correctly, I think it is fine to just reject it always. That might be a small argument for fixing it in 3.0: avoid breaking code that thinks copy=False work when you turn it into an error.

sorry - I honestly dont know what half this means...

[ajfriend]

This bug is affecting matplotlib code. Computing a colormap on a Series object has the unexpected behavior that the Series object gets modified.

[jorisvandenbossche]

We will include a fix for this in pandas 2.3 (which is targetted for in 2-3 weeks)