Skip to content

Commit 08609d9

Browse files
lwasserlwasser
committed
Phew! so many comments - more edits and cleanup pyproj page
1 parent 9ca37b2 commit 08609d9

File tree

5 files changed

+310
-256
lines changed

5 files changed

+310
-256
lines changed

‎package-structure-code/complex-python-package-builds.md

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,30 @@
11
# Complex Python package builds
22

3+
This guide is focused on packages that are either pure-python or that
4+
have a few simple extensions in another language such as C or C++.
5+
6+
If your package is more complex, [you may want to refer to this guide
7+
created by Ralf Gommers on Python packaging.](https://pypackaging-native.github.io/)
8+
9+
## Pure Python Packages vs. packages with extensions in other languages
10+
11+
You can classify Python package complexity into three general categories. These
12+
categories can in turn help you select the correct package front-end and
13+
back end tools.
14+
15+
1. **Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
16+
has the features that you want and be done with your decision!
17+
2. **Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as `C` or `C++`). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
18+
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back end of choice. We will discuss this at a high level on the complex builds page.
19+
3.**Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
20+
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
21+
22+
23+
<!--
24+
On this page, we will focus on using front-end tools to package pure python
25+
packages. We will note if a package does have the flexibility to support other
26+
back-ends and in turn more complex builds (*mentioned in #2 and #3 above*). -->
27+
<!--
328
## COmbine the two sets of statement below...
429
ELI:
530
PDM supports C/Cython extensions too: https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
@@ -86,9 +111,7 @@ CORRECTIONS:
86111
pdm doesn't use plugins. Hatch does.
87112
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
88113
89-
-->
90114
91-
```{note}
92115
??
93116
Poetry supports extensions written in other languages but this functionality is
94117
currently undocumented.
@@ -101,7 +124,6 @@ package builds.
101124
Some front-end packaging tools, such as PDM, allow you to use other
102125
build back-ends such as **meson** and **scikit-build**.
103126
104-
```
105127
106128
me:
107129
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back end.
@@ -118,3 +140,4 @@ pdm and poetry both rely on setuptools for C extensions. pdm's support claims to
118140
119141
120142
https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
143+
-->

‎package-structure-code/intro.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ the `src` directory.
9595
Intro <self>
9696
9797
Python package structure <python-package-structure>
98+
pyproject.toml Package Metadata <pyproject-toml-python-package-metadata>
9899
What are SDist & Wheel Files? <python-package-distribution-files-sdist-wheel>
99100
Package Build Tools <python-package-build-tools>
100101
Complex Builds <complex-python-package-builds>

‎package-structure-code/pyproject-toml-python-package-metadata.md

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Use a pyproject.toml file for your package configuration & metadata
2+
3+
The standard file that Python packages use to specify build requirements and
4+
metadata is called a pyproject.toml. The pyproject.toml file has become the
5+
standard for declaring Python package metadata (including dependencies) rather
6+
than using a setup.py file (or a setup.py + setup.cfg file).
7+
8+
As such you should try to [include all project based metadata and build system specifications in a `pyproject.toml` file.](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/) Using setup.py to manage both package set up and
9+
hold metadata [can cause problems with package development.](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html)
10+
11+
12+
```{admonition} Benefits of using a pyproject.toml file
13+
:class: tip
14+
15+
1. Because setup.py has a mixture of code and metadata, it will be run twice when
16+
building your package. First it will be run to extract metadata (dependencies). Then it will be run to build your package.
17+
1. Including your package's metadata in a separate human-readable `pyproject.toml` format also allows someone to view the project's metadata without
18+
running any Python code.
19+
```
20+
21+
A pyproject.toml is written in [TOML (Tom's Obvious, Minimal Language) format](https://toml.io/en/). TOML is an easy-to-read structure that is founded on key: value pairs.
22+
23+
Each section in the pyproject.toml file contains a `[table identifier]`.
24+
Below that table identifier are key value pairs that
25+
support configuration for that particular table.
26+
27+
<!-- setup.cfg for project metadata is being deprecated - set setuptools guide and
28+
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
29+
pypa -
30+
https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ -->
31+
32+
```{note}
33+
<!-- [PEP518 describes the move away from setup.py to the pyproject.toml file.](https://peps.python.org/pep-0518/) -->
34+
Python package standards are moving away from including both package metadata
35+
and Python code needed to set up a package in the same **setup.py** file.
36+
Instead we are moving towards using a **proproject.toml** file.
37+
38+
In some cases where a build is complex, a **setup.py** file may still be
39+
required. While this guide will not cover complex builds, we will provide
40+
resources working with complex builds in the future.
41+
42+
<!-- https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1071541329
43+
ELI: A complex build could mean running a python script that processes some data file and produces a pure python module file.
44+
45+
Probably not common in the scientific community specifically, but I've seen quite a few setup.py files that contain custom build stages which e.g. build gettext locale catalogs.
46+
47+
The main point is that it is more "complex" than simply copying files or directories as-is into the built wheel.
48+
-->
49+
```
50+
51+
## Example pyproject.toml
52+
53+
Below is an example build configuration for a Python project. This setup
54+
requires:
55+
56+
* **setuptools** to create the package structure,
57+
* **wheel** which is used by `setuptools` to create the [**.whl** (wheel) file](https://realpython.com/python-wheels/).
58+
* **setuptools build** to "build" the package
59+
* **setuptools_scm** to manage package version updates
60+
61+
In the example below `[build-system]` is the first table
62+
of values. It has two keys that specify the build front end and backend for a package:
63+
64+
1. `requires =`
65+
1. `build-backend =`
66+
67+
```
68+
[build-system]
69+
requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]
70+
build-backend = "setuptools.build_meta"
71+
72+
[project]
73+
name = "examplePy"
74+
authors = [
75+
{name = "Some Maintainer", email = "[email protected]"}
76+
]
77+
maintainers = [{name = "All the contributors"}]
78+
license = {text = "BSD 3-Clause"}
79+
description = "An example Python package used to support Python packaging tutorials"
80+
keywords = ["pyOpenSci", "python packaging"]
81+
readme = "README.md"
82+
83+
dependencies = [
84+
"dependency-package-name-1",
85+
"dependency-package-name-2",
86+
]
87+
```
88+
89+
90+
Notice that you also can specify dependencies in this file.
91+
92+
93+
A major benefit of the pyproject.toml file is that it makes is transparent
94+
95+
1. what build system you are using to create your package
96+
2. what dependencies you need
97+
98+
99+
The package metadata including authors, keywords, etc is also easy to read.
100+
Below you can see the same toml file that uses a different build system (PDM).
101+
Notice how simple it is to swap out the tools needed to build this package!
102+
103+
```
104+
[build-system]
105+
requires = ["pdm-pep517>=1.0.0"]
106+
build-backend = "pdm.pep517.api"
107+
108+
[project]
109+
name = "examplePy"
110+
authors = [
111+
{name = "Some Maintainer", email = "[email protected]"}
112+
]
113+
maintainers = [{name = "All the contributors"}]
114+
license = {text = "BSD 3-Clause"}
115+
description = "An example Python package used to support Python packaging tutorials"
116+
keywords = ["pyOpenSci", "python packaging"]
117+
readme = "README.md"
118+
119+
dependencies = [
120+
"dependency-package-name-1",
121+
"dependency-package-name-2",
122+
]
123+
```
124+
125+
126+
127+
```{note}
128+
[Click here to read about our packaging documentation requirements.](/package-structure-code/python-package-build-tools)
129+
```
130+
131+
<!-- TODO: add link to section on build tools when it exists and
132+
turn this into button:
133+
134+
We discuss build tools for python package more here.
135+
-->
136+
137+
138+
139+
<!-- TODO:
140+
1. add some links to packages that are using a purely toml config
141+
1. link to our example package once it's further along
142+
-->

‎package-structure-code/python-package-build-tools.md

Lines changed: 18 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -9,26 +9,19 @@ highlight tools that currently support packages with C/C++ and other language
99
extensions.
1010

1111

12-
```{admonition} Pure Python Packages vs. packages with extensions in other languages
12+
<!-- TODO: create build tool selection diagram - https://www.canva.com/design/DAFawXrierc/O7DTnqW5fmMEZ31ao-TK9w/edit -->
1313

14-
You can classify Python package complexity into three general categories. These
15-
categories can in turn help you select the correct package front-end and
16-
back end tools.
1714

18-
1. **Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
19-
has the features that you want and be done with your decision!
20-
2. **Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as `C` or `C++`). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
21-
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back end of choice. We will discuss this at a high level on the complex builds page.
22-
3. **Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
23-
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
15+
:::{figure-md} fig-target
2416

25-
On this page, we will focus on using front-end tools to package pure python
26-
packages. We will note if a package does have the flexibility to support other
27-
back-ends and in turn more complex builds (*mentioned in #2 and #3 above*).
17+
<img src="../images/python-package-tools-decision-tree.png" alt="Figure showing... will finish this once we are all happy with the figure and it's not going to change more..." width="700px">
2818

29-
[If you are interested in tool support for non pure python builds, check out this
30-
page for resources.](complex-python-package-builds)
31-
```
19+
Diagram showing the various from end build tools that you can select from. Each tool has different features as highlighted below.
20+
NOTE: this is still a DRAFT so i'm not going to spend time truly cleaning it up until i get lots of feedback on the general approach!!
21+
:::
22+
23+
If you want to know more about Python packages that have extensions written in
24+
other languages, [check out the page on complex package builds.](complex-python-package-builds)
3225

3326
## Build front-end vs. build back-end tools
3427

@@ -99,12 +92,12 @@ Example build steps using setuptools:
9992
A packaging front-end tool refers to a tool that makes it easier for you to
10093
perform common packaging tasks using similar commands. These tasks include:
10194

102-
* [Creating a Sdist and Wheel distribution](python-package-distribution-files-sdist-wheel)
103-
* Managing an environment or multiple environments in which you need to run tests and develop your package
104-
* Building documentation
95+
* [Build your packages (create the SDist and Wheel distributions](python-package-distribution-files-sdist-wheel)
10596
* Installing your package in a development mode (so it updates when you update your code)
106-
* Running tests
10797
* Publishing to PyPI
98+
* Running tests
99+
* Building documentation
100+
* Managing an environment or multiple environments in which you need to run tests and develop your package
108101

109102
There are several Python packaging tools that you can use for pure Python
110103
builds. Each front-end tool discussed below supports a slightly different set of Python
@@ -131,13 +124,13 @@ hatch build
131124
# Example to publish to PyPI:
132125
hatch publish --repository testpypi
133126
```
134-
Example build steps using **setuptools** and **build**:
127+
Example build steps using the **setuptools** backend and **build**:
135128

136129
```bash
137130
# Build the package
138131
python3 -m build
139132

140-
# Publish to test PyPI
133+
# Publish to test PyPI using twine
141134
twine upload -r testpypi dist/*
142135
```
143136

@@ -197,7 +190,7 @@ The Python developers survey results (n=>8,000 PyPI users) show setuptools and p
197190

198191
The tools that we review below include:
199192

200-
* setuptools + twine, build
193+
* Twine, Build + setuptools
201194
* Flit
202195
* Hatch
203196
* PDM
@@ -219,15 +212,6 @@ workflow to support such extensions. It also supports other backends such as sci
219212
NOTE: You can also use Hatch but you will need to write your own plugin for this support.
220213

221214

222-
<!-- TODO: create build tool selection diagram - https://www.canva.com/design/DAFawXrierc/O7DTnqW5fmMEZ31ao-TK9w/edit -->
223-
224-
225-
:::{figure-md} fig-target
226-
227-
<img src="../images/python-package-tools-decision-tree.png" alt="ADD ME." width="700px">
228-
229-
ADD ME
230-
:::
231215

232216
<!-- ### Build tools for Python packages with complex build steps
233217
If your package is not pure Python, or it has complex build steps (or build
@@ -430,7 +414,7 @@ is currently undocumented. Thus we don't recommend it for more complex builds.
430414
:widths: 20,5,50
431415
432416
Dependency management,✅,Poetry helps you add dependencies to your `pyproject.toml` metadata. _NOTE: currently Poetry adds dependencies using an approach that is slightly out of alignment with current Python peps - however there is a plan to fix this in an upcoming release._ Allows you to organize dependencies in groups: docs; package; tests.
433-
Dependency pinning,✖✅ ,Poetry offers dependency pinning however, it does so in a way that can be problematic for some packages. Read below for more.
417+
Dependency pinning,✖✅ ,Poetry offers dependency pinning however it does so in a way that can be problematic for some packages. Read below for more.
434418
Select your environment manager of choice (conda; venv; etc),✅ , Poetry allows you to either use its simple environment management tool or select the environment manager that you want to use for managing your package. [Read more about its built in environment management options](https://python-poetry.org/docs/basic-usage/#using-your-virtual-environment).
435419
Publish to PyPI and test PyPI,✅,Poetry supports publishing to both test PyPI and PyPI
436420
Version Control based versioning,✅ , The plugin (Poetry dynamic versioning)[https://github.com/mtkennerly/poetry-dynamic-versioning] supports versioning using git tags with Poetry.
@@ -473,7 +457,7 @@ This approach also won't support over ways of versioning tools, for instance,
473457
some tools use [calver](https://calver.org/) which creates new versions based on the date.
474458
```
475459

476-
```{admonition} where does this belong?
460+
```{admonition} Hatch vs PDM vs Poetry
477461
:class: note
478462
There are some features that Hatch and PDM offer that Poetry does not.
479463

0 commit comments

Comments
 (0)