Skip to content

Bump meson #57895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 17 commits into from
Closed

Bump meson #57895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d402d08
version bump
WillAyd Mar 18, 2024
668f74c
Bump meson to 0.14.0
WillAyd Mar 18, 2024
ea08a12
Fix build warnings/errors
WillAyd Mar 18, 2024
bd4b57c
more fixes
WillAyd Mar 18, 2024
a279e4a
try fix?
WillAyd Mar 18, 2024
5cb0305
more fixes
WillAyd Mar 18, 2024
97e3f50
fix macOS meson build
WillAyd Mar 18, 2024
e7dbf18
Merge remote-tracking branch 'upstream/main' into bump-meson
WillAyd Mar 19, 2024
8cfbd7f
Linux fixes
WillAyd Mar 19, 2024
184ecca
32 bit fix
WillAyd Mar 19, 2024
f2a13b6
Change around warning levels
WillAyd Mar 19, 2024
8ef0ee2
Merge remote-tracking branch 'upstream/main' into bump-meson
WillAyd Mar 19, 2024
a7ae1be
darwin fix
WillAyd Mar 19, 2024
5487fd0
Add arguments at once
WillAyd Mar 19, 2024
7def60a
meson fix
WillAyd Mar 19, 2024
bc61810
less windows warn
WillAyd Mar 19, 2024
f3a4d71
Disable MSVC warnings
WillAyd Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
- run: |
/opt/python/cp311-cp311/bin/python -m venv ~/virtualenvs/pandas-dev
. ~/virtualenvs/pandas-dev/bin/activate
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.13.1 meson[ninja]==1.2.1
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.15.0 meson[ninja]==1.4.0
python -m pip install --no-cache-dir versioneer[toml] cython numpy python-dateutil pytz pytest>=7.3.2 pytest-xdist>=2.2.0 hypothesis>=6.46.1
python -m pip install --no-cache-dir --no-build-isolation -e . --config-settings=setup-args="--werror"
python -m pip list --no-cache-dir
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ jobs:
run: |
/opt/python/cp311-cp311/bin/python -m venv ~/virtualenvs/pandas-dev
. ~/virtualenvs/pandas-dev/bin/activate
python -m pip install --no-cache-dir -U pip wheel setuptools meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --no-cache-dir -U pip wheel setuptools meson[ninja]==1.4.0 meson-python==0.15.0
python -m pip install numpy --config-settings=setup-args="-Dallow-noblas=true"
python -m pip install --no-cache-dir versioneer[toml] cython python-dateutil pytz pytest>=7.3.2 pytest-xdist>=2.2.0 hypothesis>=6.46.1
python -m pip install --no-cache-dir --no-build-isolation -e . --config-settings=setup-args="--werror"
Expand Down Expand Up @@ -265,7 +265,7 @@ jobs:
run: |
/opt/python/cp311-cp311/bin/python -m venv ~/virtualenvs/pandas-dev
. ~/virtualenvs/pandas-dev/bin/activate
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.13.1 meson[ninja]==1.2.1
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.15.0 meson[ninja]==1.4.0
python -m pip install --no-cache-dir versioneer[toml] cython numpy python-dateutil pytz pytest>=7.3.2 pytest-xdist>=2.2.0 hypothesis>=6.46.1
python -m pip install --no-cache-dir --no-build-isolation -e . --config-settings=setup-args="--werror"
python -m pip list --no-cache-dir
Expand Down Expand Up @@ -337,7 +337,7 @@ jobs:
- name: Build Environment
run: |
python --version
python -m pip install --upgrade pip setuptools wheel meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --upgrade pip setuptools wheel meson[ninja]==1.4.0 meson-python==0.15.0
python -m pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy
python -m pip install versioneer[toml]
python -m pip install python-dateutil pytz tzdata cython hypothesis>=6.46.1 pytest>=7.3.2 pytest-xdist>=2.2.0 pytest-cov
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-310.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-311-downstream_compat.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-311-numpydev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ dependencies:

# build dependencies
- versioneer[toml]
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0
- cython>=0.29.33

# test dependencies
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-311-pyarrownightly.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ dependencies:

# build dependencies
- versioneer[toml]
- meson[ninja]=1.2.1
- meson[ninja]=1.4.0
- cython>=0.29.33
- meson-python=0.13.1
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-311.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-312.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-39-minimum_versions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-39.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/actions-pypy-39.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions ci/deps/circle-310-arm64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython>=0.29.33
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
4 changes: 2 additions & 2 deletions environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ dependencies:
# build dependencies
- versioneer[toml]
- cython~=3.0.5
- meson[ninja]=1.2.1
- meson-python=0.13.1
- meson[ninja]=1.4.0
- meson-python=0.15.0

# test dependencies
- pytest>=7.3.2
Expand Down
20 changes: 17 additions & 3 deletions meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@ project(
'c', 'cpp', 'cython',
version: run_command(['generate_version.py', '--print'], check: true).stdout().strip(),
license: 'BSD-3',
meson_version: '>=1.2.1',
meson_version: '>=1.4.0',
default_options: [
'buildtype=release',
'c_std=c11',
'warning_level=2',
]
)

Expand All @@ -17,7 +16,6 @@ py = import('python').find_installation(pure: false)
tempita = files('generate_pxi.py')
versioneer = files('generate_version.py')


add_project_arguments('-DNPY_NO_DEPRECATED_API=0', language : 'c')
add_project_arguments('-DNPY_NO_DEPRECATED_API=0', language : 'cpp')

Expand All @@ -27,6 +25,22 @@ add_project_arguments('-DNPY_NO_DEPRECATED_API=0', language : 'cpp')
add_project_arguments('-DNPY_TARGET_VERSION=NPY_1_21_API_VERSION', language : 'c')
add_project_arguments('-DNPY_TARGET_VERSION=NPY_1_21_API_VERSION', language : 'cpp')

if meson.get_compiler('c').get_id() == 'msvc'
# MSVC throws a lot of C4244 warnings around loss of data with warning_level=2
# On other compilers we would have to add -Wconversion to see these
# these expose a lot of potential issues in our code base
add_project_arguments(['/w'], language: 'c')
add_project_arguments(['/w'], language: 'cpp')
else
args = ['-Wall', '-Wextra']
if build_machine.system() == 'darwin'
# error: pragma diagnostic pop could not pop, no matching push
args += ['-Wno-unknown-pragmas']
endif
add_project_arguments(args, language: 'c')
add_project_arguments(args, language: 'cpp')
endif


if fs.exists('_version_meson.py')
py.install_sources('_version_meson.py', subdir: 'pandas')
Expand Down
10 changes: 5 additions & 5 deletions pandas/_libs/byteswap.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ from libc.string cimport memcpy

def read_float_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):
cdef uint32_t value
assert offset + sizeof(value) < len(data)
assert offset + sizeof(value) < <size_t>len(data)
cdef const void *ptr = <unsigned char*>(data) + offset
memcpy(&value, ptr, sizeof(value))
if byteswap:
Expand All @@ -28,7 +28,7 @@ def read_float_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):

def read_double_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):
cdef uint64_t value
assert offset + sizeof(value) < len(data)
assert offset + sizeof(value) < <size_t>len(data)
cdef const void *ptr = <unsigned char*>(data) + offset
memcpy(&value, ptr, sizeof(value))
if byteswap:
Expand All @@ -41,7 +41,7 @@ def read_double_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):

def read_uint16_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):
cdef uint16_t res
assert offset + sizeof(res) < len(data)
assert offset + sizeof(res) < <size_t>len(data)
memcpy(&res, <const unsigned char*>(data) + offset, sizeof(res))
if byteswap:
res = _byteswap2(res)
Expand All @@ -50,7 +50,7 @@ def read_uint16_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):

def read_uint32_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):
cdef uint32_t res
assert offset + sizeof(res) < len(data)
assert offset + sizeof(res) < <size_t>len(data)
memcpy(&res, <const unsigned char*>(data) + offset, sizeof(res))
if byteswap:
res = _byteswap4(res)
Expand All @@ -59,7 +59,7 @@ def read_uint32_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):

def read_uint64_with_byteswap(bytes data, Py_ssize_t offset, bint byteswap):
cdef uint64_t res
assert offset + sizeof(res) < len(data)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning about comparison of operands with different signs (size_t is unsigned, Py_ssize_t is signed)

assert offset + sizeof(res) < <size_t>len(data)
memcpy(&res, <const unsigned char*>(data) + offset, sizeof(res))
if byteswap:
res = _byteswap8(res)
Expand Down
12 changes: 10 additions & 2 deletions pandas/_libs/hashtable_func_helper.pxi.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,13 @@ cdef value_count_{{dtype}}(const {{dtype}}_t[:] values, bint dropna, const uint8
# Don't use Py_ssize_t, since table.n_buckets is unsigned
khiter_t k

{{c_type}} val
# without explicit initialization the compiler detects that val could
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is extremely wonky, but the compiler is right that val would be uninitialized if n==0. I think it would take a decent refactor to make Cython generate something else that the compiler would be comfortable with

# be uninitialized towards the end of this function
{{if dtype == 'object'}}
{{c_type}} val = None
{{else}}
{{c_type}} val = {{to_c_type}}(0)
{{endif}}

int ret = 0
bint uses_mask = mask is not None
Expand Down Expand Up @@ -105,7 +111,7 @@ cdef value_count_{{dtype}}(const {{dtype}}_t[:] values, bint dropna, const uint8
cdef:
int64_t[::1] result_counts = np.empty(table.size + na_add, dtype=np.int64)

for i in range(table.size):
for i in range(<Py_ssize_t>table.size): # cast required for 32 bit builds
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We alternately could change table.size to be of type Py_ssize_t but leaving that to a separate exercise

{{if dtype == 'object'}}
k = kh_get_{{ttype}}(table, result_keys.data[i])
{{else}}
Expand All @@ -114,6 +120,8 @@ cdef value_count_{{dtype}}(const {{dtype}}_t[:] values, bint dropna, const uint8
result_counts[i] = table.vals[k]

if na_counter > 0:
if n <= 0: # in this case val would be uninitialized
raise ValueError("Expected n > 0")
result_counts[table.size] = na_counter
result_keys.append(val)

Expand Down
4 changes: 1 addition & 3 deletions pandas/_libs/include/pandas/vendored/klib/khash_python.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,9 @@ static void *traced_calloc(size_t num, size_t size) {
}

static void *traced_realloc(void *old_ptr, size_t size) {
PyTraceMalloc_Untrack(KHASH_TRACE_DOMAIN, (uintptr_t)old_ptr);
void *ptr = realloc(old_ptr, size);
if (ptr != NULL) {
if (old_ptr != ptr) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gcc was complaing about a use-after-free with old_ptr here

PyTraceMalloc_Untrack(KHASH_TRACE_DOMAIN, (uintptr_t)old_ptr);
}
PyTraceMalloc_Track(KHASH_TRACE_DOMAIN, (uintptr_t)ptr, size);
}
return ptr;
Expand Down
6 changes: 4 additions & 2 deletions pandas/_libs/lib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ except ImportError:

@cython.wraparound(False)
@cython.boundscheck(False)
def memory_usage_of_objects(arr: object[:]) -> int64_t:
def memory_usage_of_objects(arr: ndarray[object]) -> int64_t:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This produces:

  pandas/_libs/lib.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/lib.pyx.c:90806:12: warning: '__pyx_memview_set_object' defined but not used [-Wunused-function]
  90806 | static int __pyx_memview_set_object(const char *itemp, PyObject *obj) {
        |            ^~~~~~~~~~~~~~~~~~~~~~~~
  pandas/_libs/lib.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/lib.pyx.c:90801:20: warning: '__pyx_memview_get_object' defined but not used [-Wunused-function]
  90801 |   static PyObject *__pyx_memview_get_object(const char *itemp) {
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~

although I am not sure why it wouldn't let me specify the ndim here.

"""
Return the memory usage of an object array in bytes.

Expand Down Expand Up @@ -3058,7 +3058,9 @@ def to_object_array_tuples(rows: object) -> np.ndarray:

@cython.wraparound(False)
@cython.boundscheck(False)
def fast_multiget(dict mapping, object[:] keys, default=np.nan) -> "ArrayLike":
def fast_multiget(dict mapping,
ndarray[object, ndim=1] keys,
default=np.nan) -> "ArrayLike":
cdef:
Py_ssize_t i, n = len(keys)
object val
Expand Down
14 changes: 7 additions & 7 deletions pandas/_libs/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,18 +61,18 @@ subdir('tslibs')
libs_sources = {
# Dict of extension name -> dict of {sources, include_dirs, and deps}
# numpy include dir is implicitly included
'algos': {'sources': ['algos.pyx', _algos_common_helper, _algos_take_helper], 'deps': _khash_primitive_helper_dep},
'algos': {'sources': ['algos.pyx', _algos_common_helper, _algos_take_helper], 'deps': [_khash_primitive_helper_dep]},
'arrays': {'sources': ['arrays.pyx']},
'groupby': {'sources': ['groupby.pyx']},
'hashing': {'sources': ['hashing.pyx']},
'hashtable': {'sources': ['hashtable.pyx', _hashtable_class_helper, _hashtable_func_helper], 'deps': _khash_primitive_helper_dep},
'index': {'sources': ['index.pyx', _index_class_helper], 'deps': _khash_primitive_helper_dep},
'hashtable': {'sources': ['hashtable.pyx', _hashtable_class_helper, _hashtable_func_helper], 'deps': [_khash_primitive_helper_dep]},
'index': {'sources': ['index.pyx', _index_class_helper], 'deps': [_khash_primitive_helper_dep]},
'indexing': {'sources': ['indexing.pyx']},
'internals': {'sources': ['internals.pyx']},
'interval': {'sources': ['interval.pyx', _intervaltree_helper],
'deps': _khash_primitive_helper_dep},
'deps': [_khash_primitive_helper_dep]},
'join': {'sources': ['join.pyx', _khash_primitive_helper],
'deps': _khash_primitive_helper_dep},
'deps': [_khash_primitive_helper_dep]},
'lib': {'sources': ['lib.pyx', 'src/parser/tokenizer.c']},
'missing': {'sources': ['missing.pyx']},
'pandas_datetime': {'sources': ['src/vendored/numpy/datetime/np_datetime.c',
Expand All @@ -83,7 +83,7 @@ libs_sources = {
'src/parser/io.c',
'src/parser/pd_parser.c']},
'parsers': {'sources': ['parsers.pyx', 'src/parser/tokenizer.c', 'src/parser/io.c'],
'deps': _khash_primitive_helper_dep},
'deps': [_khash_primitive_helper_dep]},
'json': {'sources': ['src/vendored/ujson/python/ujson.c',
'src/vendored/ujson/python/objToJSON.c',
'src/vendored/ujson/python/JSONtoObj.c',
Expand Down Expand Up @@ -116,7 +116,7 @@ foreach ext_name, ext_dict : libs_sources
ext_dict.get('sources'),
cython_args: cython_args,
include_directories: [inc_np, inc_pd],
dependencies: ext_dict.get('deps', ''),
dependencies: ext_dict.get('deps', []) + [np_dep],
subdir: 'pandas/_libs',
install: true
)
Expand Down
Loading
Loading