Document CWD from where RTD execute scripts

[tswast]

Details

I'm trying to build the Python docs for the protocol buffers package, but hitting a roadblock in the installation of the protobuf package. It attempts to generate Python code for some proto files in the C++ build, but these are unavailable, as only the ./python directory appears to be copied over for the pip installation.

My .readthedocs.yml:

# .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

sphinx:
  configuration: python/docs/conf.py

# Setup build requirements for docs.
# Use conda so that we can install the latest libprotobuf package without
# having to build from scratch just for docs builds.
conda:
  environment: python/docs/environment.yml

python:
  version: 3.7
  install:
    - method: pip
      path: python

My environment.yml

name: protobuf
channels:
  - conda-forge
  - defaults
dependencies:
  - libprotobuf=3.11.2
  - make=4.2.1
  - pip=19.3.1
  - python=3.7.6
  - sphinx=2.3.1
  - sphinx_rtd_theme=0.4.3
  - sphinxcontrib-napoleon=0.7

• Read the Docs project URL: https://readthedocs.org/projects/tswast-protobuf/
• Build URL (if applicable): https://readthedocs.org/projects/tswast-protobuf/builds/10280344/
• Read the Docs username (if applicable): tswast

Expected Result

A description of what you wanted to happen

I'd like to run the pip install . / python setup.py install in-place, rather than copy the subdirectory to a tmp folder.

Actual Result

A description of what actually happened

I get file-not-found errors.

[tswast]

Note: I tried creating symlinks, but still got an error that it couldn't find the file.

https://readthedocs.org/projects/tswast-protobuf/builds/10280409/

$ pwd
/Users/swast/src/protobuf/python

$ ls -al src
total 0
drwxr-xr-x   3 swast  eng   96 Jan 17 16:36 .
drwxr-xr-x  19 swast  eng  608 Jan 17 16:39 ..
lrwxr-xr-x   1 swast  eng   42 Jan 17 16:36 descriptor.proto -> ../../src/google/protobuf/descriptor.proto

$ head src/descriptor.proto 
// Protocol Buffers - Google's data interchange format
// Copyright 2008 Google Inc.  All rights reserved.
// https://developers.google.com/protocol-buffers/
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
//     * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.

[stsewd]

I think the problem is you are assuming the cwd is python, but it's on the root of your project. You should use absolute paths using __file__ https://stackoverflow.com/questions/595305/how-do-i-get-the-path-of-the-python-script-i-am-running-in#595332

[stsewd]

Probably is worth documenting this somewhere. Probably here https://docs.readthedocs.io/en/stable/builds.html

[tswast]

I actually updated the paths to use __file__, but still got errors that the files don't exist. Are you sure that the pip install steps copy the whole repository? It seemed to copy just the python folder to a temp directory.

[stsewd]

@tswast we clone the whole repo and over that we execute the pip install ./python step. We don't copy things to the /tmp dir, looks like something from pip. You can try debugging this by executing the same steps from the build page in a clean environment.

[tswast]

Is it possible to do a pip install -e ./python? That would keep the install in-place.

[stsewd]

Currently, using the pip install option, no. But there is a workaround using a requirements file #6243 (comment)

[tswast]

Thanks!

Am I doing something wrong? I don't see the pip install -r python/docs/requirements.txt step in:

https://readthedocs.org/projects/tswast-protobuf/builds/10292724/

Added in:

tswast/protobuf@26e80c7

Maybe I can't use conda and pip in the same build? I'd love to use conda to install some C++ dependencies rather than build them from scratch. And yes, they are actually needed because the _pb2.py files aren't checked into the repo, they're generated at install time in setup.py.

[tswast]

Or maybe there's a way to run setup.py develop or conda develop?

[stsewd]

oh, when you use conda, you can't use some options from the python block #6509.

Is there a way of installing from a requirements file from conda?

[tswast]

I'll do some digging.

[stsewd]

Or maybe there's a way to run setup.py develop or conda develop?

We don't support those neither, but I'll attach those to the other issue, seems like something we could add.

[stsewd]

Couldn't find any docs about this. But looks like conda just pass the package name to pip. So in theory this should work:

dependencies:
  - pip:
    - "-r requirements.txt"

[tswast]

Success! https://readthedocs.org/projects/tswast-protobuf/builds/10309411/

Looks like I still need to update the output paths for protoc, but the docs at least build now. The trick was using setuptools instead of pip as the install method.

python:
  install:
    - method: setuptools
      path: python

pip must be copying the directory to a tmp/ folder, whereas setuptools doesn't.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.