Skip to content

Commit ac9f130

Browse files
lithomas1lithomas1
authored
Merge branch 'main' into reduce-wheel-size
2 parents 573c05c + e637b42 commit ac9f130

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+615
-144
lines changed

‎.github/workflows/unit-tests.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -241,12 +241,14 @@ jobs:
241241
fi
242242
- name: Build environment and Run Tests
243243
# https://github.com/numpy/numpy/issues/24703#issuecomment-1722379388
244+
# Note: Pinned to Cython 3.0.10 to avoid numerical instability in 32-bit environments
245+
# https://github.com/pandas-dev/pandas/pull/61423
244246
run: |
245247
/opt/python/cp313-cp313/bin/python -m venv ~/virtualenvs/pandas-dev
246248
. ~/virtualenvs/pandas-dev/bin/activate
247249
python -m pip install --no-cache-dir -U pip wheel setuptools meson[ninja]==1.2.1 meson-python==0.13.1
248250
python -m pip install numpy -Csetup-args="-Dallow-noblas=true"
249-
python -m pip install --no-cache-dir versioneer[toml] cython python-dateutil pytest>=7.3.2 pytest-xdist>=3.4.0 hypothesis>=6.84.0
251+
python -m pip install --no-cache-dir versioneer[toml] cython==3.0.10 python-dateutil pytest>=7.3.2 pytest-xdist>=3.4.0 hypothesis>=6.84.0
250252
python -m pip install --no-cache-dir --no-build-isolation -e . -Csetup-args="--werror"
251253
python -m pip list --no-cache-dir
252254
PANDAS_CI=1 python -m pytest -m 'not slow and not network and not clipboard and not single_cpu' pandas --junitxml=test-data.xml

‎.pre-commit-config.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ ci:
1919
skip: [pyright, mypy]
2020
repos:
2121
- repo: https://github.com/astral-sh/ruff-pre-commit
22-
rev: v0.11.4
22+
rev: v0.11.8
2323
hooks:
2424
- id: ruff
2525
args: [--exit-non-zero-on-fix]
@@ -95,14 +95,14 @@ repos:
9595
- id: sphinx-lint
9696
args: ["--enable", "all", "--disable", "line-too-long"]
9797
- repo: https://github.com/pre-commit/mirrors-clang-format
98-
rev: v20.1.0
98+
rev: v20.1.3
9999
hooks:
100100
- id: clang-format
101101
files: ^pandas/_libs/src|^pandas/_libs/include
102102
args: [-i]
103103
types_or: [c, c++]
104104
- repo: https://github.com/trim21/pre-commit-mirror-meson
105-
rev: v1.7.2
105+
rev: v1.8.0
106106
hooks:
107107
- id: meson-fmt
108108
args: ['--inplace']

‎Dockerfile

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,20 @@
11
FROM python:3.10.8
22
WORKDIR /home/pandas
33

4-
RUN apt-get update && apt-get -y upgrade
5-
RUN apt-get install -y build-essential bash-completion
4+
RUN apt-get update && \
5+
apt-get --no-install-recommends -y upgrade && \
6+
apt-get --no-install-recommends -y install \
7+
build-essential \
8+
bash-completion \
9+
# hdf5 needed for pytables installation
10+
libhdf5-dev \
11+
# libgles2-mesa needed for pytest-qt
12+
libgles2-mesa-dev && \
13+
rm -rf /var/lib/apt/lists/*
614

7-
# hdf5 needed for pytables installation
8-
# libgles2-mesa needed for pytest-qt
9-
RUN apt-get install -y libhdf5-dev libgles2-mesa-dev
10-
11-
RUN python -m pip install --upgrade pip
1215
COPY requirements-dev.txt /tmp
13-
RUN python -m pip install -r /tmp/requirements-dev.txt
16+
RUN python -m pip install --no-cache-dir --upgrade pip && \
17+
python -m pip install --no-cache-dir -r /tmp/requirements-dev.txt
1418
RUN git config --global --add safe.directory /home/pandas
1519

1620
ENV SHELL="/bin/bash"

‎ci/deps/actions-310-minimum_versions.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,12 @@ dependencies:
2626

2727
# optional dependencies
2828
- beautifulsoup4=4.12.3
29-
- blosc=1.21.3
3029
- bottleneck=1.3.6
3130
- fastparquet=2024.2.0
32-
- fsspec=2024.2.0
31+
- fsspec=2023.12.2
3332
- html5lib=1.1
3433
- hypothesis=6.84.0
35-
- gcsfs=2024.2.0
34+
- gcsfs=2023.12.2
3635
- jinja2=3.1.3
3736
- lxml=4.9.2
3837
- matplotlib=3.8.3
@@ -43,14 +42,15 @@ dependencies:
4342
- openpyxl=3.1.2
4443
- psycopg2=2.9.6
4544
- pyarrow=10.0.1
45+
- pyiceberg=0.7.1
4646
- pymysql=1.1.0
4747
- pyqt=5.15.9
4848
- pyreadstat=1.2.6
4949
- pytables=3.8.0
5050
- python-calamine=0.1.7
5151
- pytz=2023.4
5252
- pyxlsb=1.0.10
53-
- s3fs=2024.2.0
53+
- s3fs=2023.12.2
5454
- scipy=1.12.0
5555
- sqlalchemy=2.0.0
5656
- tabulate=0.9.0

‎ci/deps/actions-310.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ dependencies:
2424

2525
# optional dependencies
2626
- beautifulsoup4>=4.12.3
27-
- blosc>=1.21.3
2827
- bottleneck>=1.3.6
2928
- fastparquet>=2024.2.0
30-
- fsspec>=2024.2.0
29+
- fsspec>=2023.12.2
3130
- html5lib>=1.1
3231
- hypothesis>=6.84.0
33-
- gcsfs>=2024.2.0
32+
- gcsfs>=2023.12.2
3433
- jinja2>=3.1.3
3534
- lxml>=4.9.2
3635
- matplotlib>=3.8.3
@@ -41,18 +40,19 @@ dependencies:
4140
- openpyxl>=3.1.2
4241
- psycopg2>=2.9.6
4342
- pyarrow>=10.0.1
43+
- pyiceberg>=0.7.1
4444
- pymysql>=1.1.0
4545
- pyqt>=5.15.9
4646
- pyreadstat>=1.2.6
4747
- pytables>=3.8.0
4848
- python-calamine>=0.1.7
4949
- pytz>=2023.4
5050
- pyxlsb>=1.0.10
51-
- s3fs>=2024.2.0
51+
- s3fs>=2023.12.2
5252
- scipy>=1.12.0
5353
- sqlalchemy>=2.0.0
5454
- tabulate>=0.9.0
55-
- xarray>=2024.1.1, <=2024.9.0
55+
- xarray>=2024.1.1
5656
- xlrd>=2.0.1
5757
- xlsxwriter>=3.2.0
5858
- zstandard>=0.22.0

‎ci/deps/actions-311-downstream_compat.yaml

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,12 @@ dependencies:
2525

2626
# optional dependencies
2727
- beautifulsoup4>=4.12.3
28-
- blosc>=1.21.3
2928
- bottleneck>=1.3.6
3029
- fastparquet>=2024.2.0
31-
- fsspec>=2024.2.0
30+
- fsspec>=2023.12.2
3231
- html5lib>=1.1
3332
- hypothesis>=6.84.0
34-
- gcsfs>=2024.2.0
33+
- gcsfs>=2023.12.2
3534
- jinja2>=3.1.3
3635
- lxml>=4.9.2
3736
- matplotlib>=3.8.3
@@ -42,18 +41,19 @@ dependencies:
4241
- openpyxl>=3.1.2
4342
- psycopg2>=2.9.6
4443
- pyarrow>=10.0.1
44+
- pyiceberg>=0.7.1
4545
- pymysql>=1.1.0
4646
- pyqt>=5.15.9
4747
- pyreadstat>=1.2.6
4848
- pytables>=3.8.0
4949
- python-calamine>=0.1.7
5050
- pytz>=2023.4
5151
- pyxlsb>=1.0.10
52-
- s3fs>=2024.2.0
52+
- s3fs>=2023.12.2
5353
- scipy>=1.12.0
5454
- sqlalchemy>=2.0.0
5555
- tabulate>=0.9.0
56-
- xarray>=2024.1.1, <=2024.9.0
56+
- xarray>=2024.1.1
5757
- xlrd>=2.0.1
5858
- xlsxwriter>=3.2.0
5959
- zstandard>=0.22.0
@@ -63,14 +63,12 @@ dependencies:
6363
- cftime
6464
- dask
6565
- ipython
66-
- geopandas-base
6766
- seaborn
6867
- scikit-learn
6968
- statsmodels
7069
- coverage
7170
- pandas-datareader
7271
- pyyaml
73-
- py
7472
- pip:
7573
- adbc-driver-postgresql>=0.10.0
7674
- adbc-driver-sqlite>=0.8.0

‎ci/deps/actions-311.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ dependencies:
2424

2525
# optional dependencies
2626
- beautifulsoup4>=4.12.3
27-
- blosc>=1.21.3
2827
- bottleneck>=1.3.6
2928
- fastparquet>=2024.2.0
30-
- fsspec>=2024.2.0
29+
- fsspec>=2023.12.2
3130
- html5lib>=1.1
3231
- hypothesis>=6.84.0
33-
- gcsfs>=2024.2.0
32+
- gcsfs>=2023.12.2
3433
- jinja2>=3.1.3
3534
- lxml>=4.9.2
3635
- matplotlib>=3.8.3
@@ -42,17 +41,18 @@ dependencies:
4241
- openpyxl>=3.1.2
4342
- psycopg2>=2.9.6
4443
- pyarrow>=10.0.1
44+
- pyiceberg>=0.7.1
4545
- pymysql>=1.1.0
4646
- pyreadstat>=1.2.6
4747
- pytables>=3.8.0
4848
- python-calamine>=0.1.7
4949
- pytz>=2023.4
5050
- pyxlsb>=1.0.10
51-
- s3fs>=2024.2.0
51+
- s3fs>=2023.12.2
5252
- scipy>=1.12.0
5353
- sqlalchemy>=2.0.0
5454
- tabulate>=0.9.0
55-
- xarray>=2024.1.1, <=2024.9.0
55+
- xarray>=2024.1.1
5656
- xlrd>=2.0.1
5757
- xlsxwriter>=3.2.0
5858
- zstandard>=0.22.0

‎ci/deps/actions-312.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ dependencies:
2424

2525
# optional dependencies
2626
- beautifulsoup4>=4.12.3
27-
- blosc>=1.21.3
2827
- bottleneck>=1.3.6
2928
- fastparquet>=2024.2.0
30-
- fsspec>=2024.2.0
29+
- fsspec>=2023.12.2
3130
- html5lib>=1.1
3231
- hypothesis>=6.84.0
33-
- gcsfs>=2024.2.0
32+
- gcsfs>=2023.12.2
3433
- jinja2>=3.1.3
3534
- lxml>=4.9.2
3635
- matplotlib>=3.8.3
@@ -42,17 +41,18 @@ dependencies:
4241
- openpyxl>=3.1.2
4342
- psycopg2>=2.9.6
4443
- pyarrow>=10.0.1
44+
- pyiceberg>=0.7.1
4545
- pymysql>=1.1.0
4646
- pyreadstat>=1.2.6
4747
- pytables>=3.8.0
4848
- python-calamine>=0.1.7
4949
- pytz>=2023.4
5050
- pyxlsb>=1.0.10
51-
- s3fs>=2024.2.0
51+
- s3fs>=2023.12.2
5252
- scipy>=1.12.0
5353
- sqlalchemy>=2.0.0
5454
- tabulate>=0.9.0
55-
- xarray>=2024.1.1, <=2024.9.0
55+
- xarray>=2024.1.1
5656
- xlrd>=2.0.1
5757
- xlsxwriter>=3.2.0
5858
- zstandard>=0.22.0

‎ci/deps/actions-313.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,10 @@ dependencies:
2727
- blosc>=1.21.3
2828
- bottleneck>=1.3.6
2929
- fastparquet>=2024.2.0
30-
- fsspec>=2024.2.0
30+
- fsspec>=2023.12.2
3131
- html5lib>=1.1
3232
- hypothesis>=6.84.0
33-
- gcsfs>=2024.2.0
33+
- gcsfs>=2023.12.2
3434
- jinja2>=3.1.3
3535
- lxml>=4.9.2
3636
- matplotlib>=3.8.3
@@ -48,11 +48,11 @@ dependencies:
4848
- python-calamine>=0.1.7
4949
- pytz>=2023.4
5050
- pyxlsb>=1.0.10
51-
- s3fs>=2024.2.0
51+
- s3fs>=2023.12.2
5252
- scipy>=1.12.0
5353
- sqlalchemy>=2.0.0
5454
- tabulate>=0.9.0
55-
- xarray>=2024.1.1, <=2024.9.0
55+
- xarray>=2024.1.1
5656
- xlrd>=2.0.1
5757
- xlsxwriter>=3.2.0
5858
- zstandard>=0.22.0

‎doc/cheatsheet/Pandas_Cheat_Sheet.pdf

63.4 KB
Binary file not shown.

‎doc/cheatsheet/Pandas_Cheat_Sheet.pptx

187 KB
Binary file not shown.

‎doc/cheatsheet/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ This cheat sheet, originally written by Irv Lustig, [Princeton Consultants](http
1212
| Pandas_Cheat_Sheet_JA | Japanese | <a href="https://github.com/pandas-dev/pandas/blob/main/doc/cheatsheet/Pandas_Cheat_Sheet_JA.pdf" target="_parent"><img src="https://img.shields.io/badge/Open in PDF-%23FF0000.svg?style=flat-square&logo=adobe&logoColor=white"/></a> | <a href="https://github.com/pandas-dev/pandas/blob/main/doc/cheatsheet/Pandas_Cheat_Sheet_JA.pptx" target="_parent"><img src="https://img.shields.io/badge/Open in PPT-B7472A?style=flat-square&logo=microsoft-powerpoint&logoColor=white"/></a> |
1313
| Pandas_Cheat_Sheet_FA | Persian | <a href="https://github.com/pandas-dev/pandas/blob/main/doc/cheatsheet/Pandas_Cheat_Sheet_FA.pdf" target="_parent"><img src="https://img.shields.io/badge/Open in PDF-%23FF0000.svg?style=flat-square&logo=adobe&logoColor=white"/></a> | <a href="https://github.com/pandas-dev/pandas/blob/main/doc/cheatsheet/Pandas_Cheat_Sheet_FA.pptx" target="_parent"><img src="https://img.shields.io/badge/Open in PPT-B7472A?style=flat-square&logo=microsoft-powerpoint&logoColor=white"/></a> |
1414

15-
15+
The English version has additional material that is not in the versions in other languages.
1616

1717
**Alternative**
1818

‎doc/source/getting_started/install.rst

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -299,16 +299,16 @@ Dependency Minimum Versi
299299
Other data sources
300300
^^^^^^^^^^^^^^^^^^
301301

302-
Installable with ``pip install "pandas[hdf5, parquet, feather, spss, excel]"``
302+
Installable with ``pip install "pandas[hdf5, parquet, iceberg, feather, spss, excel]"``
303303

304304
====================================================== ================== ================ ==========================================================
305305
Dependency Minimum Version pip extra Notes
306306
====================================================== ================== ================ ==========================================================
307307
`PyTables <https://github.com/PyTables/PyTables>`__ 3.8.0 hdf5 HDF5-based reading / writing
308-
`blosc <https://github.com/Blosc/c-blosc>`__ 1.21.3 hdf5 Compression for HDF5; only available on ``conda``
309308
`zlib <https://github.com/madler/zlib>`__ hdf5 Compression for HDF5
310309
`fastparquet <https://github.com/dask/fastparquet>`__ 2024.2.0 - Parquet reading / writing (pyarrow is default)
311310
`pyarrow <https://github.com/apache/arrow>`__ 10.0.1 parquet, feather Parquet, ORC, and feather reading / writing
311+
`PyIceberg <https://py.iceberg.apache.org/>`__ 0.7.1 iceberg Apache Iceberg reading
312312
`pyreadstat <https://github.com/Roche/pyreadstat>`__ 1.2.6 spss SPSS files (.sav) reading
313313
`odfpy <https://github.com/eea/odfpy>`__ 1.4.1 excel Open document format (.odf, .ods, .odt) reading / writing
314314
====================================================== ================== ================ ==========================================================
@@ -329,10 +329,10 @@ Installable with ``pip install "pandas[fss, aws, gcp]"``
329329
============================================ ================== =============== ==========================================================
330330
Dependency Minimum Version pip extra Notes
331331
============================================ ================== =============== ==========================================================
332-
`fsspec <https://github.com/fsspec>`__ 2024.2.0 fss, gcp, aws Handling files aside from simple local and HTTP (required
332+
`fsspec <https://github.com/fsspec>`__ 2023.12.2 fss, gcp, aws Handling files aside from simple local and HTTP (required
333333
dependency of s3fs, gcsfs).
334-
`gcsfs <https://github.com/fsspec/gcsfs>`__ 2024.2.0 gcp Google Cloud Storage access
335-
`s3fs <https://github.com/fsspec/s3fs>`__ 2024.2.0 aws Amazon S3 access
334+
`gcsfs <https://github.com/fsspec/gcsfs>`__ 2023.12.2 gcp Google Cloud Storage access
335+
`s3fs <https://github.com/fsspec/s3fs>`__ 2023.12.2 aws Amazon S3 access
336336
============================================ ================== =============== ==========================================================
337337

338338
Clipboard

‎doc/source/getting_started/intro_tutorials/includes/titanic.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ This tutorial uses the Titanic data set, stored as CSV. The data
1111
consists of the following data columns:
1212

1313
- PassengerId: Id of every passenger.
14-
- Survived: Indication whether passenger survived. ``0`` for yes and ``1`` for no.
14+
- Survived: Indication whether passenger survived. ``0`` for no and ``1`` for yes.
1515
- Pclass: One out of the 3 ticket classes: Class ``1``, Class ``2`` and Class ``3``.
1616
- Name: Name of passenger.
1717
- Sex: Gender of passenger.

‎doc/source/reference/arrays.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -664,6 +664,7 @@ Data type introspection
664664
api.types.is_datetime64_dtype
665665
api.types.is_datetime64_ns_dtype
666666
api.types.is_datetime64tz_dtype
667+
api.types.is_dtype_equal
667668
api.types.is_extension_array_dtype
668669
api.types.is_float_dtype
669670
api.types.is_int64_dtype

‎doc/source/reference/io.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,15 @@ Parquet
156156
read_parquet
157157
DataFrame.to_parquet
158158

159+
Iceberg
160+
~~~~~~~
161+
.. autosummary::
162+
:toctree: api/
163+
164+
read_iceberg
165+
166+
.. warning:: ``read_iceberg`` is experimental and may change without warning.
167+
159168
ORC
160169
~~~
161170
.. autosummary::

0 commit comments

Comments
 (0)