COMPAT: Adapt to Numpy 2.0 dtype changes

[lithomas1]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[lithomas1]

cc @seberg for any objections to the approach here.

(I hope we aren't abusing numpy internals here).

[mattip]

I think the migration guide suggests using #include <numpy/npy_2_compat.h> and then the code change would use this macro, which includes any legacy/new dtype mangling.

- return (((PyArray_DatetimeDTypeMetaData *)dtype->c_metadata)->meta);
+ return PyDataType_C_METADATA(dtype);

But what you have should also work.

[seberg]

This is OK. I think slightly nicer would be to define #define PyDataType_C_METADATA(descr) ((descr)->c_metadata) instead and use that instead.
(That way you also don't need to vendor npy2_compat.h which is a bit much for just this purpose.

I thought there would be an elsize use in the json reader, but nice if not!

[mattip]

I think slightly nicer would be to define ...

so the complete diff would be like this?

- return (((PyArray_DatetimeDTypeMetaData *)dtype->c_metadata)->meta);
+ #if NPY_ABI_VERSION < 0x02000000
+     #define PyDataType_C_METADATA(descr) ((descr)->c_metadata)
+ #endif
+ return (PyDataType_C_METADATA(descr)->meta);

[lithomas1]

Huh, PyDataType_C_METADATA doesn't seem to exist in the ndarraytypes.h include file.
(I think that's why I copied the legacydtype thing from numpy's datetime.c file yesterday).

Does it need to be exposed on the numpy side, or am I just missing something obvious?

[lithomas1]

Maybe its something to do with the static inline declaration?

[seberg]

Hummmm, weird. Any chance there is some caching involved in CI and it is using an older NumPy wheel?

[jorisvandenbossche]

I had the same issue locally earlier today, and that was not because of caching I think (I tried with removing my pandas build directory multiple times)

[lithomas1]

No, I have it locally too

Here's what the header looks like (ndarraytypes.h)

#define PyDataType_ISLEGACY(dtype) ((dtype)->type_num < NPY_VSTRING && ((dtype)->type_num >= 0))
#define PyDataType_ISBOOL(obj) PyTypeNum_ISBOOL(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISUNSIGNED(obj) PyTypeNum_ISUNSIGNED(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISSIGNED(obj) PyTypeNum_ISSIGNED(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISINTEGER(obj) PyTypeNum_ISINTEGER(((PyArray_Descr*)(obj))->type_num )
#define PyDataType_ISFLOAT(obj) PyTypeNum_ISFLOAT(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISNUMBER(obj) PyTypeNum_ISNUMBER(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISSTRING(obj) PyTypeNum_ISSTRING(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISCOMPLEX(obj) PyTypeNum_ISCOMPLEX(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISFLEXIBLE(obj) PyTypeNum_ISFLEXIBLE(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISDATETIME(obj) PyTypeNum_ISDATETIME(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISUSERDEF(obj) PyTypeNum_ISUSERDEF(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISEXTENDED(obj) PyTypeNum_ISEXTENDED(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_ISOBJECT(obj) PyTypeNum_ISOBJECT(((PyArray_Descr*)(obj))->type_num)
#define PyDataType_MAKEUNSIZED(dtype) ((dtype)->elsize = 0)
/*
 * PyDataType_* FLAGS, FLACHK, REFCHK, HASFIELDS, HASSUBARRAY, UNSIZED,
 * SUBARRAY, NAMES, FIELDS, C_METADATA, and METADATA require version specific
 * lookup and are defined in npy_2_compat.h.
 */


#define PyArray_ISBOOL(obj) PyTypeNum_ISBOOL(PyArray_TYPE(obj))
#define PyArray_ISUNSIGNED(obj) PyTypeNum_ISUNSIGNED(PyArray_TYPE(obj))
#define PyArray_ISSIGNED(obj) PyTypeNum_ISSIGNED(PyArray_TYPE(obj))
#define PyArray_ISINTEGER(obj) PyTypeNum_ISINTEGER(PyArray_TYPE(obj))

[lithomas1]

The whitespace is where the definition of the static things should be, but they're just gone.

We still have macros like PyDataType_ISLEGACY that you added in numpy/numpy#25802 though.

[jorisvandenbossche]

And I was certainly using an up to date numpy, because wasn't using a nightly wheel but locally built from latest main. For the changes in pyarrow, I did not run into that error. I was trying to see the difference between both, but couldn't directly see anything.

[seberg]

Sorry I hadn't noticed: the definitions are moved, and you need #include "numpy/ndarrayobject.h" and import the NumPy API. I had added a new import helper to just do that locally as a quick fix if needed, but that requires copying in npy2_compat.h.

I'll have a look at it.

[seberg]

Sorry, I'll need a bit more time to set up the env for NumPy 2, but this diff should be right, I think. The annoying part is that we need to use import_array(), but it seems there is a place for this already here.

diff --git a/pandas/_libs/src/datetime/pd_datetime.c b/pandas/_libs/src/datetime/pd_datetime.c
index 19de51be6e..f778d06607 100644
--- a/pandas/_libs/src/datetime/pd_datetime.c
+++ b/pandas/_libs/src/datetime/pd_datetime.c
@@ -20,6 +20,8 @@ This file is derived from NumPy 1.7. See NUMPY_LICENSE.txt
 #include <Python.h>
 
 #include "datetime.h"
+/* Need to import_array for np_datetime.c (for NumPy 1.x support) */
+#include "numpy/ndarrayobject.h"
 #include "pandas/datetime/pd_datetime.h"
 #include "pandas/portable.h"
 
@@ -255,5 +257,6 @@ static struct PyModuleDef pandas_datetimemodule = {
 
 PyMODINIT_FUNC PyInit_pandas_datetime(void) {
   PyDateTime_IMPORT;
+  import_array();
   return PyModuleDef_Init(&pandas_datetimemodule);
 }
diff --git a/pandas/_libs/src/vendored/numpy/datetime/np_datetime.c b/pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
index 277d01807f..d2853b4936 100644
--- a/pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
+++ b/pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
@@ -16,7 +16,7 @@ This file is derived from NumPy 1.7. See NUMPY_LICENSE.txt
 
 // Licence at LICENSES/NUMPY_LICENSE
 
-#define NO_IMPORT
+#define NO_IMPORT_ARRAY
 
 #ifndef NPY_NO_DEPRECATED_API
 #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
@@ -25,7 +25,7 @@ This file is derived from NumPy 1.7. See NUMPY_LICENSE.txt
 #include <Python.h>
 
 #include "pandas/vendored/numpy/datetime/np_datetime.h"
-#include <numpy/ndarraytypes.h>
+#include <numpy/ndarrayobject.h>
 #include <numpy/npy_common.h>
 
 #if defined(_WIN32)

[lithomas1]

Sorry, I'll need a bit more time to set up the env for NumPy 2, but this diff should be right, I think. The annoying part is that we need to use import_array(), but it seems there is a place for this already here.

Thanks, I'll test this out.

Thinking out loud, maybe a simpler way could just be to use the newly exposed datetime.c functions from here https://github.com/numpy/numpy/pull/21199/files.

Would there be any ABI problems with using numpy C API functions from a new version?

[seberg]

I opened gh-57780 to iterate myself quickly, probably done, but lets see what crops up next...

Since they are proper API Functions, if you use them you cannot support NumPy 1.x unfortunately.

[lithomas1]

closing this then.

Thanks for helping debug this!