Skip to content

The virtual environment does not have the system packages as expected #10500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jnikula opened this issue Jun 29, 2023 · 7 comments · Fixed by #10562
Closed

The virtual environment does not have the system packages as expected #10500

jnikula opened this issue Jun 29, 2023 · 7 comments · Fixed by #10562
Assignees
@humitos
Labels
Needed: documentation Documentation is required

Comments

@jnikula
Copy link

jnikula commented Jun 29, 2023

Details

Expected Result

I'm installing python3-clang using apt, enabling python.system_packages, and expect my Sphinx extension to be able to import the package.

build:
  os: ubuntu-22.04
  tools:
    python: "3.11"
  apt_packages:
    - python3-clang

python:
  install:
    - method: pip
      path: .
  system_packages: true

The build log shows the system package was installed, and --system-site-packages was passed to the virtual environment.

Actual Result

Import fails with ModuleNotFoundError: No module named 'clang'.
@humitos
Copy link
Member

humitos commented Jun 29, 2023

@jnikula is there a way to install clang via pip or similar?

I don't think python.system_packages is working fine and I think we were wanting to deprecate it because it's not compatible with build.os key.

@jnikula
Copy link
Author

jnikula commented Jun 29, 2023

@jnikula is there a way to install clang via pip or similar?

@humitos Well, there's an unofficial clang package on pip... but it requires a compatible libclang to be installed on the system, and you can't get that via pip. I was hoping to dodge the compatibility issue by installing both libclang and its python bindings via apt.

I actually started to experiment with this because the current unofficial clang package is incompatible with the libclang which happens to be around on the default os environment. I guess I could try installing a specific libclang version using apt and a specific python clang version using pip, and hope for the best.

I don't think python.system_packages is working fine and I think we were wanting to deprecate it because it's not compatible with build.os key.

Just curious, what's the incompatibility there? Obviously it depends on what's available in the os, which may be surprising in some cases.

@humitos
Copy link
Member

humitos commented Jun 29, 2023

I guess I could try installing a specific libclang version using apt and a specific python clang version using pip, and hope for the best.

I'd say this is a good approach to try here, actually 👍🏼

Just curious, what's the incompatibility there? Obviously it depends on what's available in the os, which may be
surprising in some cases.

The build system does not uses the Python version from the system itself, but it uses one compiled with asdf instead. So, all the Python libraries installed in the system won't be available in the version compiled with asdf.

@humitos humitos added the Needed: more information A reply from issue author is required label Jun 29, 2023
jnikula added a commit to jnikula/hawkmoth that referenced this issue Jun 29, 2023
@jnikula
doc: avoid python.system_packages=true in .readthedocs.yaml
59ad9b2 
Apparently python.system_packages=true is problematic in the RTD
environment [1]. Install libclang-14-dev from apt and clang from pip,
using specific versions for both.

Attempt to install clang using build.jobs.post_install instead of a
requirements.txt to avoid defining the version in two different
files. Fingers crossed.

[1] readthedocs/readthedocs.org#10500
@jnikula
Copy link
Author

jnikula commented Jun 29, 2023 •
edited
Loading

I guess I could try installing a specific libclang version using apt and a specific python clang version using pip, and hope for the best.

I'd say this is a good approach to try here, actually 👍🏼

Thanks, this worked.

The build system does not uses the Python version from the system itself, but it uses one compiled with asdf instead. So, all the Python libraries installed in the system won't be available in the version compiled with asdf.

Okay, thanks for the explanation!

I noticed that https://blog.readthedocs.com/migrate-configuration-v2/ actually says "Use system packages is deprecated in favor of declaring all your dependencies via a requirements file using python.requirements.", however the actual configuration file specification does not mention it's deprecated: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-system-packages

Maybe consider this report to be about amending the config file spec, then?

@humitos
Copy link
Member

humitos commented Jun 29, 2023

however the actual configuration file specification does not mention it's deprecated: docs.readthedocs.io/en/stable/config-file/v2.html#python-system-packages

I noticed this today. We should update the documentation to mention this is deprecated as well 👍🏼 . Thanks.

@humitos humitos added Needed: documentation Documentation is required and removed Needed: more information A reply from issue author is required labels Jul 24, 2023
humitos added a commit that referenced this issue Jul 24, 2023
@humitos
Docs: do not mention python.system_packages
9275a56 
This option doesn't work on builds using `build.os` because the Python binary
used it's not the one from the system, but another one compiled via `asdf`.
This means that creating a virtualenv with access to the "system packages" has
no effect at all.

As a first step, I'm removing this option from the documentation to avoid
confusions. Next, we should probably our deprecation policy to contact users
ands  remove it from the code as well.

Closes #10500
@humitos humitos mentioned this issue Jul 24, 2023
Merged
@github-project-automation github-project-automation bot moved this to Planned in 📍Roadmap Aug 3, 2023
@humitos humitos moved this from Planned to In progress in 📍Roadmap Aug 3, 2023
@humitos humitos self-assigned this Aug 3, 2023
@humitos humitos moved this from In progress to Needs review in 📍Roadmap Aug 3, 2023
@humitos
Copy link
Member

humitos commented Aug 10, 2023

I wanted to comment here that we are deprecating system_packages. Read more about at https://blog.readthedocs.com/drop-support-system-packages/

@jnikula
Copy link
Author

jnikula commented Aug 10, 2023

Please add the information to https://docs.readthedocs.io/en/stable/config-file/v2.html#python-system-packages too!

humitos added a commit that referenced this issue Aug 22, 2023
@humitos @ericholscher @agjohnson
Deprecation: remove "use system packages" (python.system_packages c…
bcac962 
…onfig key and UI checkbox) (#10562)

* Docs: do not mention `python.system_packages`

This option doesn't work on builds using `build.os` because the Python binary
used it's not the one from the system, but another one compiled via `asdf`.
This means that creating a virtualenv with access to the "system packages" has
no effect at all.

As a first step, I'm removing this option from the documentation to avoid
confusions. Next, we should probably our deprecation policy to contact users
ands  remove it from the code as well.

Closes #10500

* Build: remove "use system package" option

- Removed from the UI
- Removed from the v1 and v2 config file

* Do not remove the DB field for now

We should remove this field after the deploy.

* Remove `use_system_site_packages` leftovers

* Display custom error messages based on the config key

* Apply suggestions from code review

Co-authored-by: Eric Holscher <[email protected]>

* Minor code style changes

* Update readthedocs/config/config.py

Co-authored-by: Anthony <[email protected]>

---------

Co-authored-by: Eric Holscher <[email protected]>
Co-authored-by: Anthony <[email protected]>
@github-project-automation github-project-automation bot moved this from Needs review to Done in 📍Roadmap Aug 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@humitos humitos

Labels
Needed: documentation Documentation is required
Projects
📍Roadmap
Archived in project
Milestone
No milestone
2 participants
@humitos @jnikula