PipEnv Development dependencies not included in requirements.txt

[heri16]

Behavior of pipenv lock -r has been changed.
See: pypa/pipenv#942

We now require another pipenv lock -r -d to produce a requirements.txt that includes dev dependencies required for the build.

An example would be pipenv --dev Cython

[heri16]

After reading the discussion in here, pypa/pipenv#972 , the API behavior might change sometime in the future. We need a future proof way of fixing this like in ONSdigital/eq-survey-runner#1327

[heri16]

@dschep Should Cython be a dev dependency? Wished pipenv had an option to include it as a build dependency.

I would suggest that the correct build behavior going forward should be the following:

pipenv lock --requirements > requirements.txt
pipenv lock --dev > requirements-dev.txt

pip install -r requirements-dev.txt
pip install -r requirements.txt
pip uninstall -r requirements-dev.txt
echo 'Zip artifacts...'

[dschep]

Hmm. yeah, as you saw in #87 we don't want to include dev deps in the deployment. I haven't used Cython personally, if I understand right, it needs to installed when pip install is called, is that right?

[heri16]

Yup. Cython needs to be installed first before doing pip install of other packages that detect the presence of Cython (before compiling optimizations automatically).

Example:
pip install cython
pip install cymysql
pip uninstall cython

[dschep]

Cool, thanks. This is pretty tricky to address afaict. dev reqs with pipenv don't seem to be the right fit, since i'm just using pipenv lock -r and you don't really want to deploy cython anyway. Also, we'd want build deps to work somehow with requirements.txt based projects too. Tho if you're fine with the dev reqs getting deployed, I think it'd be reasonable to add an option to this plugin to add the -d flag to the pipenv lock call.

As a work around, my suggestions would be:

• if not using docker: create&activate a virtualenv and install cython
• if using docker: create your own docker image that has cython installed

[heri16]

"create your own docker image that has cython installed" seems like a good workaround, since the docker file would only need three lines. I would say this could be added to the documentation to address build dependencies.

Or even better, a custom config option could be added to specify additional commands to be run in the docker container before the actual build. For example to yum install some devel libraries. That would make it very user friendly:

• the user does not have to learn docker and Dockerfile,
• the user does not have to maintain the docker image, ensuring the image is up-to-date with the upstream over the long run.

The commands could be read from serverless.yml and placed into ".serverless/pre-build-hook.sh"

[dschep]

lol.. how could I forget I started on that in #103.. but it needs to be adapted to work for docker correctly.

[pgrzesik]

Hey 👋 I'm closing this ticket as it looks like it's heavily outdated, we can of course reopen it if needed 👍