(Review 2 - complete) Packaging section - part 2

.github/workflows/artifact_redirect.yml

@@ -0,0 +1,18 @@
+name: Book Preview
+
+on: [status]
+
+jobs:
+  circleci_artifacts_redirector_job:
+    runs-on: ubuntu-latest
+    if: "${{ github.event.context == 'ci/circleci: build_book' }}"
+    name: Run CircleCI artifacts redirector
+    steps:
+      - name: GitHub Action step
+        id: step1
+        uses: larsoner/circleci-artifacts-redirector-action@master
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          artifact-path: 0/html/index.html
+          circleci-jobs: build_book
+          job-title: Click to preview rendered book

conf.py

@@ -17,12 +17,12 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'python-package-guide'
-copyright = '2023, pyOpenSci'
-author = 'pyOpenSci Community'
+project = "python-package-guide"
+copyright = "2023, pyOpenSci"
+author = "pyOpenSci Community"
 
 # The full version, including alpha/beta/rc tags
-release = '0.1'
+release = "0.1"
 
 # -- General configuration ---------------------------------------------------
 
@@ -47,7 +47,7 @@
 myst_heading_anchors = 3
 
 # For generating sitemap
-html_baseurl = 'https://www.pyopensci.org/software-peer-review/'
+html_baseurl = "https://www.pyopensci.org/software-peer-review/"
 
 # Link to our repo for easy PR/ editing
 html_theme_options = {
@@ -81,7 +81,7 @@
     "header_links_before_dropdown": 4,
     "use_edit_page_button": True,
     "show_toc_level": 1,
-    #"navbar_align": "left",  # [left, content, right] For testing that the navbar items align properly
+    # "navbar_align": "left",  # [left, content, right] For testing that the navbar items align properly
     "github_url": "https://github.com/pyopensci/python-package-guide",
     "twitter_url": "https://twitter.com/pyopensci",
     "footer_items": ["copyright"],
@@ -99,11 +99,11 @@
 
 # Add analytics to furo theme
 gtagjs_ids = [
-    'UA-141260825-1',
+    "UA-141260825-1",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -114,20 +114,20 @@
     ".DS_Store",
     ".github",
     ".nox",
-    "README.md"
-    ]
+    "README.md",
+]
 
 # For sitemap
-html_baseurl = 'https://www.pyopensci.org/package-review-guide/'
+html_baseurl = "https://www.pyopensci.org/package-review-guide/"
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'pydata_sphinx_theme'
+html_theme = "pydata_sphinx_theme"
 html_static_path = ["_static"]
 html_css_files = ["pyos.css"]
+html_title = "pyOpenSci Python Packaging Guide"
 html_js_files = ["matomo.js"]
-html_title = "pyOpenSci Package Guide"
 html_logo = "images/logo/logo.png"

index.md

@@ -1,18 +1,19 @@
 # pyOpenSci Python Open Source Package Development Guide
 
-
 ```{toctree}
 :hidden:
 :caption: Documentation
 
 Documentation <documentation/index>
+
 ```
 
 ```{toctree}
 :hidden:
 :caption: Packaging
 
-Packaging  <python-packaging/intro>
+Packaging <package-structure-code/intro>
+
 ```
 
 ```{toctree}
@@ -28,7 +29,8 @@ https://github.com/pyOpenSci/python-package-guide/community -->
 ## Welcome, Python open source enthusiast!
 
 Here you will find guidelines for what we look for in your scientific
-Python package when reviewing. You will also find best practice recommendations and curated lists of community resources surrounding packaging and documentation.
+Python package when reviewing. You will also find best practice recommendations and curated lists of community resources surrounding packaging and documentation. Our goal is to help the
+community make decisions around how to create scientific Python packages. We are working towards a shared vision of packaging that helps users better understand where to start.
 
 ::::{grid} 2
 :reverse:
@@ -43,7 +45,6 @@ Python package when reviewing. You will also find best practice recommendations
 :columns: 8
 :class: sd-fs-3
 
-
 ```{button-link} https://www.pyopensci.org/about-peer-review/
 :color: primary
 :class: sd-rounded-pill float-left
@@ -59,7 +60,6 @@ Learn about our open peer review process
 :::
 ::::
 
-
 <!-- I think this is the end of the header - below begins the next grid-->
 
 ::::{grid} 1 1 2 2
@@ -80,6 +80,15 @@ documentation that are
 commonly used in the scientific Python community.
 :::
 
+:::{grid-item-card}
+:link: package-structure-code/intro
+:link-type: doc
+:class-header: bg-light
+
+✨ Python packaging tools & structure ✨
+^^^
+All of the modern tools discussed in this guide will help you build an efficient packaging workflow. This section helps you select the tool that will work best for you.
+:::
 
 :::{grid-item-card}
 :link: CONTRIBUTING
@@ -94,21 +103,22 @@ contribute.
 ::::
 
 ## Who this guidebook is for
+
 We assume that you are here because you are:
 
+1. Looking for guidance on creating a Python package.
+1. Looking for resources associated with Python packaging.
 1. Considering submitting a package to pyOpenSci and want to understand what we are looking for when we review your package
-2. Looking for guidance on creating a Python package.
-3. Looking for resources associated with Python packaging.
 
 Well, friend, you've come to the right place!
 
 ## What you will find in this guidebook
 
 This guidebook contains:
 
-* Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
-* Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
-* A curated list of resources to help you get your package into documented, usable and tested shape.
+- Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
+- Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
+- A curated list of resources to help you get your package into documented, usable and tested shape.
 
 ## Where this guide is headed
 
@@ -119,9 +129,6 @@ Good meets the requirements. Going beyond the minimum can make package maintenan
 This guide is now a work in progress. If you have ideas of things you'd like
 to see here, [we invite you to open an issue on GitHub that details any changes or additions that you'd like to see.](https://github.com/pyOpenSci/python-package-guide/issues).
 
-
-
-
 <!--
 COMMENTED OUT TEXT TO BE MOVED
 

package-structure-code/complex-python-package-builds.md

@@ -0,0 +1,146 @@
+# Complex Python package builds
+
+This guide is focused on packages that are either pure-python or that
+have a few simple extensions in another language such as C or C++.
+
+In the future, we want to provide resources for packaging workflows that require more complex builds. If you have questions about these types of package, please [add a question to our discourse](https://pyopensci.discourse.group/) or open an [issue about this guide specifically in the GitHub repo for this guide](https://github.com/pyOpenSci/python-package-guide/issues). There are many nuances to building and distributing Python packages that have compiled extensions requiring non-Python dependencies at build time. For an overview and thorough discussion of these nuances, please see [this site.](https://pypackaging-native.github.io/)
+
+## Pure Python Packages vs. packages with extensions in other languages
+
+You can classify Python package complexity into three general categories. These
+categories can in turn help you select the correct package front-end and
+back-end tools.
+
+1. **Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
+   has the features that you want and be done with your decision!
+2. **Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as C or C++). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
+   supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back-end of choice. We will discuss this at a high level on the complex builds page. 3.**Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
+   or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
+
+<!--
+On this page, we will focus on using front-end tools to package pure python
+packages. We will note if a package does have the flexibility to support other
+back-ends and in turn more complex builds (*mentioned in #2 and #3 above*). -->
+<!--
+## COmbine the two sets of statement below...
+ELI:
+PDM supports C/Cython extensions too: https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
+
+It does this by allowing you to write a python script that gets injected into a setuptools build process :) so that's not necessarily the greatest choice. It's a bit like using setuptools directly. ;)
+
+Ralf:
+Hatch only supports pure Python packages as of now. setuptools is still a very reasonable choice, and okay if all you have is a few C/Cython extensions. But I'd say you should probably recommend meson-python and scikit-build-core as the two best tools for building packages containing compiled extensions.
+
+
+* link to ralf's blog and book on complex builds
+* keep this page high level so we don't get weight downsides
+* can use the examplePy repo stefan and I are working on that will test various build combinations
+
+*****
+
+ELI: It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure python files.
+
+Hatch - https://hatch.pypa.io/latest/config/build/#build-hooks uild hooks
+
+Ralf -
+Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
+****
+
+
+ HEnry: Poetry will move to PEP 621 configuration in version 2.
+
+* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
+* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook. so while some have used it with other back-end builds it's not ideal for this application
+* pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
+* hatch both offers a plugin type approach to support custom build steps
+PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
+At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
+
+
+
+**Setuptools** is the oldest tool in the above list. While it doesn't have a
+friendly user front end, because "OG" tool that has been used for Python packaging for over a decade, we discuss it here.
+
+**Hatch** and PDM are newer, more modern tool that support customization of any
+part of your packaging steps. These tools also support some C and C++
+extensions.
+
+
+OFEK - Why use hatchlin vs pdm back-end -
+File inclusion is more configurable and easier by default
+There is already a rich ecosystem of plugins and a well-thought-out interface
+Consistency since the official Python packaging tutorial uses Hatchling by default
+
+
+Henry -
+The scikit-hep cookie provides 11 back-ends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the sdist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
+
+<!-- TODO: add - compatible with other build back-ends eg pdm can work with hatchling
+
+Eli:
+poetry: supports it, but is undocumented and uses setuptools under the hood, they plan to change how this works and then document it
+pdm-back-end: supports it, and documents it -- and also uses setuptools under the hood
+hatchling: permits you to define hooks for you to write your own custom build steps, including to build C++ extensions
+
+-->
+
+<!-- from eli about pdm
+It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure Python files.
+
+Comment about hatch.
+https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1081108118
+
+From ralf: There are no silver bullets here yet, no workflow tool is complete. Both Hatch and PDM are single-author tools, which is another concern. @eli-schwartz's assessment is unfortunately correct here I believe (at a high level at least, not sure about details). Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
+
+-->
+
+<!--TODO Add examples of builds using each of the tools below?
+
+pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
+poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
+PDM and hatch both offer a plugin type approach to support custom build steps
+PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
+At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
+
+CORRECTIONS:
+pdm doesn't use plugins. Hatch does.
+pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
+
+
+??
+Poetry supports extensions written in other languages but this functionality is
+currently undocumented.
+
+Tools such as Setuptools, PDM, Hatch and Poetry all have some level of support
+for C and C++ extensions.
+Some Python packaging tools,
+such as **Flit** and the **flit-core** build back-end only support pure-Python
+package builds.
+Some front-end packaging tools, such as PDM, allow you to use other
+build back-ends such as **meson** and **scikit-build**.
+
+
+me:
+pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
+poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
+PDM and hatch both offer a plugin type approach to support custom build steps
+PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
+At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
+@eli-schwartz eli-schwartz 3 weeks ago
+PDM and hatch both offer a plugin type approach to support custom build steps
+
+ELI:
+pdm doesn't use plugins. Hatch does.
+pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
+
+
+https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
+-->
+
+<!-- https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1071541329
+ELI: A complex build could mean running a python script that processes some data file and produces a pure python module file.
+
+Probably not common in the scientific community specifically, but I've seen quite a few setup.py files that contain custom build stages which e.g. build gettext locale catalogs.
+
+The main point is that it is more "complex" than simply copying files or directories as-is into the built wheel.
+-->