Add: packaging tools & layout section to the guide 🎉

(Review 2) Packaging section - part 2 (Reviews Welcome)

Author: lwasser

.github/workflows/artifact_redirect.yml

@@ -0,0 +1,18 @@
+name: Book Preview
+
+on: [status]
+
+jobs:
+  circleci_artifacts_redirector_job:
+    runs-on: ubuntu-latest
+    if: "${{ github.event.context == 'ci/circleci: build_book' }}"
+    name: Run CircleCI artifacts redirector
+    steps:
+      - name: GitHub Action step
+        id: step1
+        uses: larsoner/circleci-artifacts-redirector-action@master
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          artifact-path: 0/html/index.html
+          circleci-jobs: build_book
+          job-title: Click to preview rendered book

.gitignore

@@ -5,3 +5,4 @@ tmp/
 .DS_Store
 .nox
 __pycache__
+*notes-from-review.md

conf.py

@@ -17,12 +17,12 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'python-package-guide'
-copyright = '2023, pyOpenSci'
-author = 'pyOpenSci Community'
+project = "python-package-guide"
+copyright = "2023, pyOpenSci"
+author = "pyOpenSci Community"
 
 # The full version, including alpha/beta/rc tags
-release = '0.1'
+release = "0.1"
 
 # -- General configuration ---------------------------------------------------
 
@@ -47,7 +47,7 @@
 myst_heading_anchors = 3
 
 # For generating sitemap
-html_baseurl = 'https://www.pyopensci.org/software-peer-review/'
+html_baseurl = "https://www.pyopensci.org/software-peer-review/"
 
 # Link to our repo for easy PR/ editing
 html_theme_options = {
@@ -81,7 +81,7 @@
     "header_links_before_dropdown": 4,
     "use_edit_page_button": True,
     "show_toc_level": 1,
-    #"navbar_align": "left",  # [left, content, right] For testing that the navbar items align properly
+    # "navbar_align": "left",  # [left, content, right] For testing that the navbar items align properly
     "github_url": "https://github.com/pyopensci/python-package-guide",
     "twitter_url": "https://twitter.com/pyopensci",
     "footer_items": ["copyright"],
@@ -99,11 +99,11 @@
 
 # Add analytics to furo theme
 gtagjs_ids = [
-    'UA-141260825-1',
+    "UA-141260825-1",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -114,20 +114,20 @@
     ".DS_Store",
     ".github",
     ".nox",
-    "README.md"
-    ]
+    "README.md",
+]
 
 # For sitemap
-html_baseurl = 'https://www.pyopensci.org/package-review-guide/'
+html_baseurl = "https://www.pyopensci.org/package-review-guide/"
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'pydata_sphinx_theme'
+html_theme = "pydata_sphinx_theme"
 html_static_path = ["_static"]
 html_css_files = ["pyos.css"]
+html_title = "pyOpenSci Python Packaging Guide"
 html_js_files = ["matomo.js"]
-html_title = "pyOpenSci Package Guide"
 html_logo = "images/logo/logo.png"

images/python-package-tools-2022-survey-pypa.png

@@ -1,18 +1,19 @@
 # pyOpenSci Python Open Source Package Development Guide
 
-
 ```{toctree}
 :hidden:
 :caption: Documentation
 
 Documentation <documentation/index>
+
 ```
 
 ```{toctree}
 :hidden:
 :caption: Packaging
 
-Packaging  <python-packaging/intro>
+Packaging <package-structure-code/intro>
+
 ```
 
 ```{toctree}
@@ -28,7 +29,8 @@ https://github.com/pyOpenSci/python-package-guide/community -->
 ## Welcome, Python open source enthusiast!
 
 Here you will find guidelines for what we look for in your scientific
-Python package when reviewing. You will also find best practice recommendations and curated lists of community resources surrounding packaging and documentation.
+Python package when reviewing. You will also find best practice recommendations and curated lists of community resources surrounding packaging and documentation. Our goal is to help the
+community make decisions around how to create scientific Python packages. We are working towards a shared vision of packaging that helps users better understand where to start.
 
 ::::{grid} 2
 :reverse:
@@ -43,7 +45,6 @@ Python package when reviewing. You will also find best practice recommendations
 :columns: 8
 :class: sd-fs-3
 
-
 ```{button-link} https://www.pyopensci.org/about-peer-review/
 :color: primary
 :class: sd-rounded-pill float-left
@@ -59,7 +60,6 @@ Learn about our open peer review process
 :::
 ::::
 
-
 <!-- I think this is the end of the header - below begins the next grid-->
 
 ::::{grid} 1 1 2 2
@@ -80,6 +80,15 @@ documentation that are
 commonly used in the scientific Python community.
 :::
 
+:::{grid-item-card}
+:link: package-structure-code/intro
+:link-type: doc
+:class-header: bg-light
+
+✨ Python packaging tools & structure ✨
+^^^
+All of the modern tools discussed in this guide will help you build an efficient packaging workflow. This section helps you select the tool that will work best for you.
+:::
 
 :::{grid-item-card}
 :link: CONTRIBUTING
@@ -94,21 +103,22 @@ contribute.
 ::::
 
 ## Who this guidebook is for
+
 We assume that you are here because you are:
 
+1. Looking for guidance on creating a Python package.
+1. Looking for resources associated with Python packaging.
 1. Considering submitting a package to pyOpenSci and want to understand what we are looking for when we review your package
-2. Looking for guidance on creating a Python package.
-3. Looking for resources associated with Python packaging.
 
 Well, friend, you've come to the right place!
 
 ## What you will find in this guidebook
 
 This guidebook contains:
 
-* Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
-* Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
-* A curated list of resources to help you get your package into documented, usable and tested shape.
+- Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
+- Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
+- A curated list of resources to help you get your package into documented, usable and tested shape.
 
 ## Where this guide is headed
 
@@ -118,38 +128,3 @@ Good meets the requirements. Going beyond the minimum can make package maintenan
 
 This guide is now a work in progress. If you have ideas of things you'd like
 to see here, [we invite you to open an issue on GitHub that details any changes or additions that you'd like to see.](https://github.com/pyOpenSci/python-package-guide/issues).
-
-
-
-
-<!--
-COMMENTED OUT TEXT TO BE MOVED
-
-
-# TODO LINK TO CI BUILDS FOR Documentation>
-Maybe we can curate a list of CI builds that people can use??? or is that moving too close to a cookie cutter situation
-
-The text below is being moved to the packaging infrastructure section which
-doesn't exist YET... but will soon .
-pyOpenSci packages must:
-
-- Contain full documentation for any user-facing functions.
-- Have a test suite that covers the major functionality of the package.
-- Use continuous integration.
-- Use an OSI approved software license.
-
-
-## Other recommendations
-### Python version support
-You should always be explicit about which versions of Python your package supports.
-Keeping compatibility with old Python versions can be difficult as functionality changes.
-A good rule of thumb is that the package should support, at least,
-the latest three Python versions (e.g., 3.8, 3.7, 3.6).
-
-### Code Style
-pyOpenSci encourages authors to consult [PEP 8](https://www.python.org/dev/peps/pep-0008/) for information on how to style your code.
-
-### Linting
-An automatic linter (e.g. flake8) can help ensure your code is clean and free of syntax errors. These can be integrated with your CI.
-
--->

images/python-package-tools-decision-tree.png

@@ -0,0 +1,18 @@
+# Complex Python package builds
+
+This guide is focused on packages that are either pure-python or that
+have a few simple extensions in another language such as C or C++.
+
+In the future, we want to provide resources for packaging workflows that require more complex builds. If you have questions about these types of package, please [add a question to our discourse](https://pyopensci.discourse.group/) or open an [issue about this guide specifically in the GitHub repo for this guide](https://github.com/pyOpenSci/python-package-guide/issues). There are many nuances to building and distributing Python packages that have compiled extensions requiring non-Python dependencies at build time. For an overview and thorough discussion of these nuances, please see [this site.](https://pypackaging-native.github.io/)
+
+## Pure Python Packages vs. packages with extensions in other languages
+
+You can classify Python package complexity into three general categories. These
+categories can in turn help you select the correct package front-end and
+back-end tools.
+
+1. **Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
+   has the features that you want and be done with your decision!
+2. **Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as C or C++). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
+   supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back-end of choice. We will discuss this at a high level on the complex builds page. 3.**Python packages that have extensions written in different languages (e.g. Fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
+   or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.

images/python-package-tools-decision-tree.svg

@@ -0,0 +1,75 @@
+# Python package structure information
+
+This section provides guidance on your Python package's structure, code formats and style. It also reviews the various packaging tools that you can use to
+support building and publishing your package.
+
+If you are confused by Python packaging, you are not alone!
+The good news is there are some great modern packaging
+tools that ensure that you're following best practices. Here, we
+review tool features and suggest tools that might be best fitted for your workflow.
+
+```{note}
+If you are considering submitting a package for peer review, have a look at the
+bare-minimum [editor checks](https://www.pyopensci.org/software-peer-review/how-to/editor-in-chief-guide.html#editor-checklist-template) that pyOpenSci
+performs before a review begins. These checks are useful to explore
+for both authors planning to submit a package to us for review and for
+anyone who is just getting started with creating a Python package.
+
+```
+
+## What you will learn here
+
+In this section of our Python packaging guide, we:
+
+- Provide an overview of the options available to you when packaging your tool
+- Suggest tools and approaches that both meet your needs and also support existing standards.
+- Suggest tools and approaches that will allow you to expand upon a workflow that may begin as a pure Python tool and evolve into a tool that requires addition layers of complexity in the packaging build.
+- Align our suggestions with the most current, accepted
+  [PEPs (Python Enhancement Protocols)](https://peps.python.org/pep-0000/) and the [scientific-python community SPECs](https://scientific-python.org/specs/).
+- In an effort to maintain consistency within our community, we also align with existing best practices being implemented by developers of core Scientific Python packages such as Numpy, SciPy and others.
+
+## Guidelines for pyOpenSci's packaging recommendations
+
+<!-- Might belong on the LANDING page for this entire guide?-->
+
+The flexibility of the Python programming language lends itself to a diverse
+range of tool options for creating a Python package. Python is so flexible that
+it is one of the few languages that can be used to wrap around other languages. The ability of Python to wrap other languages one the reasons why you will often hear Python described as a ["glue" language](https://numpy.org/doc/stable/user/c-info.python-as-glue.html)"
+
+If you are building a pure Python package, then your packaging setup can be
+simple. However, some scientific packages have complex requirements as they may
+need to support extensions or tools written in other languages such as C or C++.
+
+To support the many different uses of Python, there are many ways to create a
+Python package. In this guide, we suggest approaches for packaging approaches and tools based
+upon:
+
+1. What we think will be best and easiest to adopt for those who are newer to packaging
+2. Tools that we think are well maintained and documented.
+3. A shared goal of standardizing packaging approaches across this (scientific) Python ecosystem.
+
+Here, we also try to align our suggestions with the most current, accepted
+[Python community](https://packaging.python.org/en/latest/) and [scientific community](https://scientific-python.org/specs/).
+
+```{admonition} Suggestions in this guide are not pyOpenSci review requirements
+:class: important
+
+The suggestions for package layout in this section are made with the
+intent of being helpful; they are not specific requirements for your
+package to be reviewed and accepted into our pyOpenSci open source ecosystem.
+
+Please check out our [package scope page](https://www.pyopensci.org/software-peer-review/about/package-scope.html) and [review requirements in our author guide](https://www.pyopensci.org/software-peer-review/how-to/author-guide.html#) if you are looking for pyOpenSci's Python package review requirements!
+```
+
+```{toctree}
+:hidden:
+:caption: Package structure & code style
+
+Intro <self>
+
+Python package structure <python-package-structure>
+pyproject.toml Package Metadata <pyproject-toml-python-package-metadata>
+What are SDist & Wheel Files? <python-package-distribution-files-sdist-wheel>
+Package Build Tools <python-package-build-tools>
+Complex Builds <complex-python-package-builds>
+```