Skip to content

Commit 305d4eb

Browse files
mroeschketopper-123
mroeschke
authored and
topper-123
committed
CI: Build wheel from sdist (pandas-dev#53087)
* CI: Build wheel from sdist * Save sdist name to pass to ciwheelbuild * Fix typo * fix typo * Check dist * Another typo * one more typo? * Remove missing files in Manifest, checkout project * try a different variable * Try in the prior directory up * check project folder * Pull path * Exclude some data files * Replace test command * Remove .data as well * Install locale on musl * Try delvewheel * Run wheel unpack everywhere * Skip in script * Wheel unpack separately * Invalid toml * Run wheel unpack outside * dont create dir * Debug mac unpack, try alternative test * Align * for loop validate, try to fix -v * for loop validate, try to fix -v * Fix path * Try pwd * To root? * Use another bash? * try MSYS_NO_PATHCONV * Add backslash * Remove a slash * Use powershell * Use variable * Typo * Add other deps * Final cleanups * Remove unecessary select * Try no * Make multiline comment * Put pack repo owner * Try single * clean step title * Update to setup micromaba * Use create-args * Use create args * Set PANDAS_CI * Try running all on powershell * Use ; * Try this command * Add double? * try escpaing quotes * Escape brackets? * remove comma * Single quotes * Other quotes * single? * Backslash? * All the escapes * No marks * Fix quotes * test don't repair windows wheel * Uncomment delvewheel
1 parent 98938c9 commit 305d4eb

File tree

7 files changed

+105
-267
lines changed

7 files changed

+105
-267
lines changed

‎.github/workflows/wheels.yml

+83-122
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# In an attempt to save CI resources, wheel builds do
55
# not run on each push but only weekly and for releases.
66
# Wheel builds can be triggered from the Actions page
7-
# (if you have the perms) on a commit to master.
7+
# (if you have the permissions) on a commit to main.
88
#
99
# Alternatively, you can add labels to the pull request in order to trigger wheel
1010
# builds.
@@ -14,13 +14,8 @@ name: Wheel builder
1414

1515
on:
1616
schedule:
17-
# ┌───────────── minute (0 - 59)
18-
# │ ┌───────────── hour (0 - 23)
19-
# │ │ ┌───────────── day of the month (1 - 31)
20-
# │ │ │ ┌───────────── month (1 - 12 or JAN-DEC)
21-
# │ │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
22-
# │ │ │ │ │
23-
- cron: "27 3 */1 * *"
17+
# 3:27 UTC every day
18+
- cron: "27 3 * * *"
2419
push:
2520
pull_request:
2621
types: [labeled, opened, synchronize, reopened]
@@ -37,170 +32,136 @@ permissions:
3732
contents: read
3833

3934
jobs:
40-
build_wheels:
41-
name: Build wheel for ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
35+
build_sdist:
36+
name: Build sdist
4237
if: >-
4338
(github.event_name == 'schedule' && github.repository_owner == 'pandas-dev') ||
4439
github.event_name == 'workflow_dispatch' ||
4540
(github.event_name == 'pull_request' &&
4641
contains(github.event.pull_request.labels.*.name, 'Build')) ||
4742
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && ( ! endsWith(github.ref, 'dev0')))
48-
runs-on: ${{ matrix.buildplat[0] }}
49-
strategy:
50-
# Ensure that a wheel builder finishes even if another fails
51-
fail-fast: false
52-
matrix:
53-
# GitHub Actions doesn't support pairing matrix values together, let's improvise
54-
# https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
55-
buildplat:
56-
- [ubuntu-20.04, manylinux_x86_64]
57-
- [macos-11, macosx_*]
58-
- [windows-2019, win_amd64]
59-
# TODO: support PyPy?
60-
python: [["cp39", "3.9"], ["cp310", "3.10"], ["cp311", "3.11"]]# "pp39"]
43+
runs-on: ubuntu-22.04
6144
env:
6245
IS_PUSH: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
6346
IS_SCHEDULE_DISPATCH: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
47+
outputs:
48+
sdist_file: ${{ steps.save-path.outputs.sdist_name }}
6449
steps:
6550
- name: Checkout pandas
6651
uses: actions/checkout@v3
6752
with:
68-
submodules: true
69-
# versioneer.py requires the latest tag to be reachable. Here we
70-
# fetch the complete history to get access to the tags.
71-
# A shallow clone can work when the following issue is resolved:
72-
# https://github.com/actions/checkout/issues/338
7353
fetch-depth: 0
7454

75-
- name: Build wheels
76-
uses: pypa/[email protected]
77-
env:
78-
CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
79-
80-
# Used to test(Windows-only) and push the built wheels
81-
# You might need to use setup-python separately
82-
# if the new Python-dev version
83-
# is unavailable on conda-forge.
84-
- uses: conda-incubator/setup-miniconda@v2
55+
- name: Set up Python
56+
uses: actions/setup-python@v4
8557
with:
86-
auto-update-conda: true
87-
python-version: ${{ matrix.python[1] }}
88-
activate-environment: test
89-
channels: conda-forge, anaconda
90-
channel-priority: true
91-
# mamba fails to solve, also we really don't need this since we're just installing python
92-
# mamba-version: "*"
93-
94-
- name: Test wheels (Windows 64-bit only)
95-
if: ${{ matrix.buildplat[1] == 'win_amd64' }}
96-
shell: cmd /C CALL {0}
58+
python-version: '3.11'
59+
60+
- name: Build sdist
9761
run: |
98-
python ci/test_wheels.py wheelhouse
62+
python -m pip install build
63+
python -m build --sdist
9964
10065
- uses: actions/upload-artifact@v3
10166
with:
102-
name: ${{ matrix.python[0] }}-${{ startsWith(matrix.buildplat[1], 'macosx') && 'macosx' || matrix.buildplat[1] }}
103-
path: ./wheelhouse/*.whl
104-
67+
name: sdist
68+
path: ./dist/*
10569

106-
- name: Install anaconda client
107-
if: ${{ success() && (env.IS_SCHEDULE_DISPATCH == 'true' || env.IS_PUSH == 'true') }}
70+
- name: Output sdist name
71+
id: save-path
10872
shell: bash -el {0}
109-
run: conda install -q -y anaconda-client
73+
run: echo "sdist_name=$(ls ./dist)" >> "$GITHUB_OUTPUT"
11074

111-
112-
- name: Upload wheels
113-
if: ${{ success() && (env.IS_SCHEDULE_DISPATCH == 'true' || env.IS_PUSH == 'true') }}
114-
shell: bash -el {0}
115-
env:
116-
PANDAS_STAGING_UPLOAD_TOKEN: ${{ secrets.PANDAS_STAGING_UPLOAD_TOKEN }}
117-
PANDAS_NIGHTLY_UPLOAD_TOKEN: ${{ secrets.PANDAS_NIGHTLY_UPLOAD_TOKEN }}
118-
run: |
119-
source ci/upload_wheels.sh
120-
set_upload_vars
121-
# trigger an upload to
122-
# https://anaconda.org/scipy-wheels-nightly/pandas
123-
# for cron jobs or "Run workflow" (restricted to main branch).
124-
# Tags will upload to
125-
# https://anaconda.org/multibuild-wheels-staging/pandas
126-
# The tokens were originally generated at anaconda.org
127-
upload_wheels
128-
build_sdist:
129-
name: Build sdist
75+
build_wheels:
76+
needs: build_sdist
77+
name: Build wheel for ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
13078
if: >-
131-
github.event_name == 'schedule' ||
79+
(github.event_name == 'schedule' && github.repository_owner == 'pandas-dev') ||
13280
github.event_name == 'workflow_dispatch' ||
13381
(github.event_name == 'pull_request' &&
13482
contains(github.event.pull_request.labels.*.name, 'Build')) ||
13583
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && ( ! endsWith(github.ref, 'dev0')))
136-
runs-on: ubuntu-22.04
84+
runs-on: ${{ matrix.buildplat[0] }}
85+
strategy:
86+
fail-fast: false
87+
matrix:
88+
# GitHub Actions doesn't support pairing matrix values together, let's improvise
89+
# https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
90+
buildplat:
91+
- [ubuntu-22.04, manylinux_x86_64]
92+
- [ubuntu-22.04, musllinux_x86_64]
93+
- [macos-12, macosx_*]
94+
- [windows-2022, win_amd64]
95+
# TODO: support PyPy?
96+
python: [["cp39", "3.9"], ["cp310", "3.10"], ["cp311", "3.11"]]
13797
env:
13898
IS_PUSH: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
13999
IS_SCHEDULE_DISPATCH: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
140100
steps:
141101
- name: Checkout pandas
142102
uses: actions/checkout@v3
143103
with:
144-
submodules: true
145-
# versioneer.py requires the latest tag to be reachable. Here we
146-
# fetch the complete history to get access to the tags.
147-
# A shallow clone can work when the following issue is resolved:
148-
# https://github.com/actions/checkout/issues/338
149104
fetch-depth: 0
150105

151-
# Used to push the built sdist
152-
- uses: conda-incubator/setup-miniconda@v2
106+
- name: Download sdist
107+
uses: actions/download-artifact@v3
153108
with:
154-
auto-update-conda: true
155-
# Really doesn't matter what version we upload with
156-
# just the version we test with
157-
python-version: '3.10'
158-
channels: conda-forge
159-
channel-priority: true
160-
# mamba fails to solve, also we really don't need this since we're just installing python
161-
# mamba-version: "*"
109+
name: sdist
110+
path: ./dist
162111

163-
- name: Build sdist
164-
run: |
165-
pip install build
166-
python -m build --sdist
167-
- name: Test the sdist
168-
shell: bash -el {0}
169-
run: |
170-
# TODO: Don't run test suite, and instead build wheels from sdist
171-
# by splitting the wheel builders into a two stage job
172-
# (1. Generate sdist 2. Build wheels from sdist)
173-
# This tests the sdists, and saves some build time
174-
python -m pip install dist/*.gz
175-
pip install hypothesis>=6.46.1 pytest>=7.0.0 pytest-xdist>=2.2.0 pytest-asyncio>=0.17
176-
cd .. # Not a good idea to test within the src tree
177-
python -c "import pandas; print(pandas.__version__);
178-
pandas.test(extra_args=['-m not clipboard and not single_cpu and not slow and not network and not db', '-n 2']);
179-
pandas.test(extra_args=['-m not clipboard and single_cpu and not slow and not network and not db'])"
180-
- uses: actions/upload-artifact@v3
112+
- name: Build wheels
113+
uses: pypa/[email protected]
181114
with:
182-
name: sdist
183-
path: ./dist/*
115+
package-dir: ./dist/${{ needs.build_sdist.outputs.sdist_file }}
116+
env:
117+
CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
184118

185-
- name: Install anaconda client
186-
if: ${{ success() && (env.IS_SCHEDULE_DISPATCH == 'true' || env.IS_PUSH == 'true') }}
119+
- name: Set up Python
120+
uses: mamba-org/setup-micromamba@v1
121+
with:
122+
environment-name: wheel-env
123+
create-args: >-
124+
python=${{ matrix.python[1] }}
125+
anaconda-client
126+
wheel
127+
cache-downloads: true
128+
cache-environment: true
129+
130+
- name: Validate wheel RECORD
187131
shell: bash -el {0}
132+
run: for whl in $(ls wheelhouse); do wheel unpack wheelhouse/$whl -d /tmp; done
133+
134+
# Testing on windowsservercore instead of GHA runner to fail on missing DLLs
135+
- name: Test Windows Wheels
136+
if: ${{ matrix.buildplat[1] == 'win_amd64' }}
137+
shell: pwsh
188138
run: |
189-
conda install -q -y anaconda-client
139+
$TST_CMD = @"
140+
python -m pip install pytz six numpy python-dateutil tzdata>=2022.1 hypothesis>=6.46.1 pytest>=7.0.0 pytest-xdist>=2.2.0 pytest-asyncio>=0.17;
141+
python -m pip install --find-links=pandas\wheelhouse --no-index pandas;
142+
python -c `'import pandas as pd; pd.test()`';
143+
"@
144+
docker pull python:${{ matrix.python[1] }}-windowsservercore
145+
docker run --env PANDAS_CI='1' -v ${PWD}:C:\pandas python:${{ matrix.python[1] }}-windowsservercore powershell -Command $TST_CMD
146+
147+
- uses: actions/upload-artifact@v3
148+
with:
149+
name: ${{ matrix.python[0] }}-${{ startsWith(matrix.buildplat[1], 'macosx') && 'macosx' || matrix.buildplat[1] }}
150+
path: ./wheelhouse/*.whl
190151

191-
- name: Upload sdist
152+
- name: Upload wheels & sdist
192153
if: ${{ success() && (env.IS_SCHEDULE_DISPATCH == 'true' || env.IS_PUSH == 'true') }}
193154
shell: bash -el {0}
194155
env:
195156
PANDAS_STAGING_UPLOAD_TOKEN: ${{ secrets.PANDAS_STAGING_UPLOAD_TOKEN }}
196157
PANDAS_NIGHTLY_UPLOAD_TOKEN: ${{ secrets.PANDAS_NIGHTLY_UPLOAD_TOKEN }}
158+
# trigger an upload to
159+
# https://anaconda.org/scipy-wheels-nightly/pandas
160+
# for cron jobs or "Run workflow" (restricted to main branch).
161+
# Tags will upload to
162+
# https://anaconda.org/multibuild-wheels-staging/pandas
163+
# The tokens were originally generated at anaconda.org
197164
run: |
198165
source ci/upload_wheels.sh
199166
set_upload_vars
200-
# trigger an upload to
201-
# https://anaconda.org/scipy-wheels-nightly/pandas
202-
# for cron jobs or "Run workflow" (restricted to main branch).
203-
# Tags will upload to
204-
# https://anaconda.org/multibuild-wheels-staging/pandas
205-
# The tokens were originally generated at anaconda.org
206167
upload_wheels

‎MANIFEST.in

+7-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,3 @@
1-
include RELEASE.md
2-
include versioneer.py
3-
41
graft doc
52
prune doc/build
63

@@ -10,6 +7,7 @@ graft pandas
107

118
global-exclude *.bz2
129
global-exclude *.csv
10+
global-exclude *.data
1311
global-exclude *.dta
1412
global-exclude *.feather
1513
global-exclude *.tar
@@ -18,9 +16,12 @@ global-exclude *.h5
1816
global-exclude *.html
1917
global-exclude *.json
2018
global-exclude *.jsonl
19+
global-exclude *.kml
2120
global-exclude *.msgpack
2221
global-exclude *.pdf
22+
global-exclude *.parquet
2323
global-exclude *.pickle
24+
global-exclude *.pkl
2425
global-exclude *.png
2526
global-exclude *.pptx
2627
global-exclude *.ods
@@ -29,12 +30,15 @@ global-exclude *.orc
2930
global-exclude *.sas7bdat
3031
global-exclude *.sav
3132
global-exclude *.so
33+
global-exclude *.txt
3234
global-exclude *.xls
3335
global-exclude *.xlsb
3436
global-exclude *.xlsm
3537
global-exclude *.xlsx
3638
global-exclude *.xpt
3739
global-exclude *.cpt
40+
global-exclude *.xml
41+
global-exclude *.xsl
3842
global-exclude *.xz
3943
global-exclude *.zip
4044
global-exclude *.zst

‎ci/fix_wheels.py

-61
This file was deleted.

0 commit comments

Comments
 (0)