Install step runs python -m pip install --upgrade --upgrade-strategy eager and clobbers most of conda's installed packages

[valeriupredoi]

Hi guys, first off let me just say a big thank you for all your work here, we are happily using Readthedocs, just like many others, and it's great! Second, my apologies if this issue has been discussed elsewhere and decisions were taken, but I thought I'd bring it forward, even if we actually found a solution for it (see below).

So - a vanilla install and build configuration when needing some extra deps (ie install the package that the docs are built for) looks like this (what we had until yesterday):

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "mambaforge-4.10"

# Declare the requirements required to build your docs
conda:
  environment:
    environment.yml

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - doc

# Build documentation in the doc directory with Sphinx
sphinx:
  configuration: doc/sphinx/source/conf.py
  fail_on_warning: true

# If using Sphinx, optionally build your docs in additional formats such as PDF
formats:
  - pdf

and this worked fine for us until xgboost had an issue whereby their PyPi version was different from the conda-forge package version - quite normal during a release cycle, when usually PyPi is ahead of conda-forge by a couple days. Only then I noticed that the pip install performed by the workflow from Readthedocs is doing a full eager install of all packages, even if most of them have been installed via the conda env. This is not optimal for a few reasons - it takes longer, issues could arise from package version conflicts, pip may install a package that is literally broken ie a PyPi version may be compiled differently and not play well in a conda env, etc. So I was wondering if you could support user opts for the pip install, with a default a la pip install . Let me know if I can help in any way BTW - I tend to open issues that I also want to help with!

We have found a workaround this by using jobs - and cheers for implementing that! We are literally just pip installing editable as a post env build step:

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "mambaforge-4.10"
  jobs:
    post_create_environment:
      - pip install -e .[develop]

# Declare the requirements required to build your docs
conda:
  environment:
    environment.yml

# Build documentation in the doc directory with Sphinx
sphinx:
  configuration: doc/sphinx/source/conf.py
  fail_on_warning: true

# If using Sphinx, optionally build your docs in additional formats such as PDF
formats:
  - pdf

Many thanks! 🍺

[humitos]

Hi @valeriupredoi! Thanks for opening this issue. We have some issues related to this topic already. I found #5545 that looks related and others less related. However, it seems most of them are related to the combination of Conda + virtualenv environments for some reason 🤷🏼

In the past, we have considered allowing users to specify the arguments for pip, and also for sphinx-build command. In fact, we've added some "feature flags" some time ago for this and also a sphinx.fail_on_warnings config file option for Sphinx. However, the overhead that it requires implementing each of the possible options it's too high compared to the value they add. So, we went one step further on this topic and implemented build.jobs. It's a generic solution that cover lot of use cases and also these these special cases where projects require a slightly different build process (as you already found). Later, we implemented build.commands, that allows projects to override the default build process completely when it's required. Docs at https://docs.readthedocs.io/en/stable/build-customization.html

This approach gave users the flexibility to make their build process exactly as they need without us having to enable a feature flag for them or codifying a new config key each time a new one appears. I think this decision was a win-win: projects are able to keep building their projects without having to hack anything or wait for us to enable a "feature flag", and we can use that development and support time on keep improving other pieces of the platform 💯

I suppose that this is not the reply that you were expecting 😅 , but at least it explains a little history behind this decision. I'm super happy that you solved the problem using build.jobs because is exactly what I'd expect users to do 😉 , and what I'd have recommended you if I'd be were asked, so 👍🏼

[valeriupredoi]

@humitos very many thanks for a most detailed explanation! I opened this issue thinking that this particular case was not looked at by you guys (obviously playing Dr Obvious here, since it was bound to have been looked at 😁 ), and I completely understand your way of thinking around it - jobs do the proper trick for us and we couldn't be any happier. Cheers muchly again for the good work! Am closing this since it's served its purpose, and I learned from it too 🍺