Support for PEP 735 dependency groups

[cthoyt]

What's the problem this feature will solve?

PEP 735 introduced dependency groups, which are complementary to optional dependencies in that dependency groups might not correspond to features in the package, but rather be something like development or release dependencies.

You can install something with dependency groups like this:

$ pip install --dependency-groups=tests,typing

ReadTheDocs currently supports specifying optional dependencies (c.f., https://docs.readthedocs.io/en/stable/config-file/v2.html#packages) with configuration like the following, where docs probably has sphinx, sphinx-rtd-theme, and other sphinx plugins.

version: 2

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

Describe the solution you'd like

I'd like an additional configuration in the install dictionary that works similar to extra_requirements that would correspond to dependency groups. Let's say I have used the PEP 735 definition of my docs, like in this abbreviated pyproject.toml:

[dependency-groups]
tests = [
    "pytest",
    "coverage",
]
docs = [
    "sphinx>=8",
    "sphinx-rtd-theme>=3.0",
    "sphinx_automodapi",
]

then I would want to define it with an alternate key, like dependency_groups (just a suggestion for the name):

version: 2

python:
  install:
    - method: pip
      path: .
      dependency_groups:
        - docs

Alternative solutions

This approach isn't required to get the intended results, which is of course to install the right extra dependencies to get my docs to build. However, the main goal is to better organize my metadata, and not to expose sphinx as an "extra" on PyPI, which doesn't exactly fit the spirit of extras/optional dependencies

Another in-progress solution (mentioned in #11766 (comment)) is #11710, which will allow for:

version: 2

build:
  jobs:
    install:
      - pip install --dependency-groups=test,typing 

Additional context

This is motivated by recent improvements in pip, uv, and tox! I have an (almost) working demo in my cookiecutter project which shows how to update configuration properly to support this cthoyt/cookiecutter-snekpack#32

I am not familiar at all with the RTD stack, but if there's a way I can contribute to coding this up, please let me know :)

Nitty-gritty

Here are a few places which probably would be part of a theoretical implementation:

readthedocs.org/readthedocs/config/models.py

Lines 77 to 81 in 404d82a

	class PythonInstall(Base):
	__slots__ = (
	"path",
	"method",
	"extra_requirements",

I'd simply add a new slot dependency_groups here

readthedocs.org/readthedocs/doc_builder/python_environments.py

Lines 66 to 67 in 404d82a

	if install.extra_requirements:
	extra_req_param = "[{}]".format(",".join(install.extra_requirements))

here's how I'd update this:

# Added these next lines
dependency_group_args = []
if install.dependency_groups:
    # not clear if the equals is necessary or if
    # this can be broken into two parts
    dependency_group_args.append("--dependency-groups={}".format(",".join(install.dependency_groups))

extra_req_param = ""
if install.extra_requirements:
    extra_req_param = "[{}]".format(",".join(install.extra_requirements))
self.build_env.run(
    self.venv_bin(filename="python"),
    "-m",
    "pip",
    "install",
    "--upgrade",
    "--upgrade-strategy",
    "only-if-needed",
    "--no-cache-dir",
    "{path}{extra_requirements}".format(
        path=local_path,
        extra_requirements=extra_req_param,
    ),
    *dependency_group_args,
    cwd=self.checkout_path,
    bin_path=self.venv_bin(),
)

As a minor note, I would also do a bit of refactoring to store all of the args into the list and then splat all of them into run()

[humitos]

Hi @cthoyt, thanks for opening this issue.

I don't think we will support this soon since it requires a lot of extra work from our side and we are focusing ourselves on different features currently.

However, this use case will be a really good fit for the work we are doing on #11710. Once that PR gets merged, you will be able to write something like:

version: 2

build:
  jobs:
    install:
      - pip install --dependency-groups=test,typing 

[cthoyt]

@humitos super, I hope this brings a lot of simplification to your work!

I will be happy to test run that this is working properly when #11710 is ready.

Feel free to close or leave this issue open in case anyone else comes looking for it.

[fepegar]

Happy to test as well. I'm having the same problem (build) at this PR:

• Split extra dependencies and dependency groups TorchIO-project/torchio#1238

[agjohnson]

You should be able to test this out, #11710 was released this week and it seems to behaving well so far. We'd be curious to know how this goes and if this is an easy way to support custom/alternative installation methods.

[fepegar]

It seems that pip doesn't have --dependency-groups (yet). I would typically use uv for this. Do you have any tips?

[fepegar]

I guess a more general question is: how does one use the new feature?

[agjohnson]

Yeah we don't have any serious docs on this yet, you'll have to glue some pieces together here.

I haven't followed dependency groups in pip, but if it's a case of pip being out of date, you might also need to upgrade pip in your own project for now. The version we install might be lagging a little bit.

For uv, support isn't built in but can be achieved with some extra setup:
#11289

In the end, you should have something like:

build:
  jobs:
    install:
      - pip install --dependency-groups=tests,typing

That is just a rough example though.

[fepegar]

Thank you, @agjohnson. This seems to have worked:

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  jobs:
    install:
      - pip install .
      - pip install dependency-groups
      - pip-install-dependency-groups doc

[ewuerger]

For anyone wondering how to do it with uv.

For mkdocs:

jobs:
    create_environment:
      - asdf plugin add uv
      - asdf install uv latest
      - asdf global uv latest
      - uv sync --group docs
    build:
      html:
        - NO_COLOR=1 uv run --no-sync mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html

and for sphinx:

jobs:
    create_environment:
      - asdf plugin add uv
      - asdf install uv latest
      - asdf global uv latest
      - uv sync --group docs
    build:
      html:
        - make -C docs html BUILDDIR=$READTHEDOCS_OUTPUT

The Makefile with the html command can be seen here:
https://github.com/DSD-DBS/py-capellambse/blob/7b570178b82f3f0d96df14adc3aa4ca4199acac0/docs/Makefile
Neet setup we have in py-capellambse.

[humitos]

Thanks for sharing these examples.

I think the only actionable could be to write these examples in our documentation at https://docs.readthedocs.io/en/latest/build-customization.html

[neutrinoceros]

Isn't it already the case ?
https://docs.readthedocs.io/en/latest/build-customization.html#install-dependencies-with-uv

[humitos]

Well, yes, but that example doesn't mention/use dependency groups.

[stevepiercy]

For anyone who wants pull request previews to build with uv, I use a combination of make commands in my Makefile.

.PHONY: dev
dev:  ## Install required Python, create Python virtual environment, and install package requirements
	@uv python install ">=3.11,<3.13"
	@uv venv
	@uv sync

.PHONY: rtd-prepare
rtd-prepare:  ## Prepare environment on Read the Docs
	asdf plugin add uv
	asdf install uv latest
	asdf global uv latest

.PHONY: rtd-pr-preview
rtd-pr-preview: rtd-prepare dev ## Build pull request preview on Read the Docs
	cd $(DOCS_DIR) && $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) ${READTHEDOCS_OUTPUT}/html/

The command make rtd-pr-preview is called from my .readthedocs.yaml file.

# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the OS, Python version and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "3.13"
  commands:
    # Cancel building pull requests when there aren't changes in the docs directory or YAML file.
    # You can add any other files or directories that you'd like here as well,
    # like your docs requirements file, or other files that will change your docs build.
    #
    # If there are no changes (git diff exits with 0) we force the command to return with 183.
    # This is a special exit code on Read the Docs that will cancel the build immediately.
    - |
      if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- docs/ styles/ .readthedocs.yaml .vale.ini Makefile uv.lock \
      src/plone_sphinx_theme/theme/plone-sphinx-theme/static/styles/plone-sphinx-theme.css \
      src/plone_sphinx_theme/theme/plone-sphinx-theme/static/scripts/plone-sphinx-theme.js;
      then
        exit 183;
      fi
    - make rtd-pr-preview

[humitos]

@stevepiercy I'm a little confused with your example and I'm not sure I follow it completely. Is it using "dependency groups"? I understand it doesn't, right? We already have an example documented to use uv in that way at https://docs.readthedocs.com/platform/stable/build-customization.html#install-dependencies-with-uv

This issue is about using dependency groups either with pip or uv -- and we are missing a simple example in our docs that shows that particular case.

[stevepiercy]

@humitos no, it doesn't. I found this issue when looking for a way to use pull request previews with uv, not build the published docs. I derived my example from commands both in this issue and the docs. Sorry for hijacking the issue in my haste, but it seemed useful. Maybe we should create a new issue from my example?

[humitos]

I found this issue when looking for a way to use pull request previews with uv, not build the published docs.

Why you would have different build processes? That could be an issue, since everything could work fine in the PR, but break in production.

[stevepiercy]

@humitos sorry, I wasn't clear again. My build and PR preview processes are the same. I was only looking for examples to use, and there were none for PR previews. Maybe a link on this page to example configuration that only exists in the build docs would close the loop?

Anyway, the example configuration in your published docs uses uv pip install, whereas mine uses uv sync, which uses a uv.lock file. The following links good reads that explain why I use uv sync instead.

• What's the difference between uv pip install and uv add astral-sh/uv#9219
• https://medium.com/@jayantnehra18/managing-python-dependencies-with-uv-e6d2c52fd488