Need to install Python module that have system dependencies

[AndrewFarley]

Problem

Unable to install Python modules that have system dependencies, such as mysqlclient, and there's no faculty in this package to do so.

Suggested Solution

Add support for an optional pre-requirements shell script to be run. This shell script (which the end-user must write) must be able to both install system packages, and then copy any desired artifacts into the folder to get packaged. In my above instance, I would suggest something like...

  pythonRequirements:
    dockerizePip: true
    preReqScript: install-mysql-devel.sh

and that shell script install-mysql-devel.sh (in the same folder as serverless.yml) would contain...

#!/bin/bash
yum install -y mysql-devel
cp /usr/lib64/mysql/libmysqlclient.so.18.0.0 ./libmysqlclient.so.18

I'll put a bounty on this feature of ~$100 either in money or stuff, your choice. I need this badly, and don't have the time to make this feature right now. I will eventually implement this if no one's free because I'll need it and it will annoy me too much.

[AndrewFarley]

I suppose an alternate (maybe simpler?) way to implement this might be to allow the user to specify an array list of file(s) to gather from the host docker OS to be copied/zipped into the lambda. Then the user could simply do their own docker overlay to run the "install" script, and specify the includes to copy in serverless.yml. :) Other ideas appreciated.

[dschep]

The plugin allows you to specify your own docker image, so you can just create a simple docker file like:

FROM lambci/lambda:build-python3.6
RUN yum install -y mysql-devel

then run docker build -t lambda-python3.6-with-mysql-build-deps .
and add this to your serverless.yml:

custom:
  pythonRequirements:
    dockerImage: lambda-python3.6-with-mysql-build-deps

This handles the library being available at compile time. Then something like what you mentioned in your 2nd comment would work for gttting the file out of the image.

If you manually added libmysqlclient.so.18 to the puck, would it work? I imagine /var/task isn't on the LD_LIRBARY_PATH and you can't change that in lambda 😞

[dschep]

Oooh. you could abuse ENTRY_POINT to copy the .so file maybe, just a thought

[AndrewFarley]

Yeah, already tested, manually adding libmysqlclient and other libraries to the package works perfectly with that example script above. I'm already using a custom docker image at the moment with the system libs pre-installed as you mentioned, and have manually copied the libraries I need and committed them to the root of my codebase so they get bundled with my lambda, however I'd like to avoid doing that in the future for portability and maintainability (I don't want to maintain docker image overlays). I would be okay with someone implementing that second idea if possible to simply copy files needed, as that would suffice for most extended needs. However, I think the allowing the user to run a custom script would allow for a simple and smooth build pipeline, the user would be 100% in serverless, not having to run docker commands to prepare an image, just to be able to run serverless. Whatever would make the most sense for the future of this plugin (which I use heavily). Thoughts? :P

[dschep]

Cool. Sounds good. I'll take a stab at it tonight. As for the bounty, instead of payment to myself or @unitedincome, I'd love to accept proof of donation to the PSF instead 😃

[AndrewFarley]

Sure, don't mind donating to a worth cause as well! :) Thanks

[manav95]

I am trying to use dlib for my python project, which requires system dependencies and needs to run cmake to be installed properly. Would a custom Docker image handle that or do I need to put in the dlib dependencies and Linux binaries manually? If so, how do I do that while still retaining the plugin?

[dschep]

@manav95, I need to revisit #103 for when using docker, and once I get that fixed, you should be able to use that feature for your needs.

[manav95]

@dschep Can I still use the serverless-python-requirements plugin for most of my pip dependencies and then use a docker script for the dlib plugin? dlib is the only dependency that has this issue.

[AndrewFarley]

@manav95 You can absolutely still use the serverless python requirements for your project. What I've done temporarily is spun up my own docker container, based on the same one that this project uses, but then installed the software I need and copied any libraries (that need to be in the lambda) to my project root and made sure those files in the include: in serverless.yml. Then, I made a local image of the modified docker container, and used the name of that docker image in the serverless.yml dockerImage key you can specify. Eg:

custom:
  pythonRequirements: 
    invalidateCaches: true
    dockerizePip: true
    dockerImage: "farley/python-modified:v1"

I'm also eagerly awaiting this feature, as my main development box is not Linux. However, my CI/CD boxes are linux and don't have to use docker. So, on my CI/CD boxes I don't use the dockerize feature, but for local development/deployment I do so I don't need to run VMs. :)

[manav95]

How do I go about cloning the docker image used for this project

[dschep]

see #101 (comment)

[jadhavmanoj]

@dschep I have to add mysql in the image. I tried creating new Docker image by adding

FROM lambci/lambda:build-python3.6 RUN yum install -y mysql-devel

but got this error
Rpmdb checksum is invalid: dCDPT(pkg checksums): mysql-config.x86_64 0:5.5.57-1.18.amzn1 - u
not able to figure out why this happens.

[dschep]

It installs fine AFAICT, try updating the docker image with docker pull maybe? Another option is to just use the amazonlinux image and also install python3.6.

[maheshgawali]

Getting same issue as @jadhavmanoj , after updating the docker image as @dschep suggested, now I am getting this
Rpmdb checksum is invalid: dCDPT(pkg checksums): libsepol-devel.x86_64 0:2.1.7-3.12.amzn1 - u

is this issue related to https://forums.aws.amazon.com/thread.jspa?threadID=244745

[jadhavmanoj]

@dschep Thanks for your suggestion.
I created a custom image using amazonlinux image as a base image.

FROM library/amazonlinux
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python
RUN yum install -y mysql-devel
....

Note: there is an issue with MySQL-python package and mysqlclient package.
So I used PyMySQL==0.8.0 which solve my problem.

[dschep]

Reading that issue it sounds like I might be using a different overlay storage driver (I'm on Ubuntu). Glad you found a work around @jadhavmanoj!

[alexjurkiewicz]

It would be great to get this in the README

[brett-matthews]

Has anyone tried this with Django 2.2? Im not 100% sure, but I think Im installing mysqlclient in the correct way however all I keep seeing in CloudWatch is;

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module. Did you install mysqlclient?

I have Dockerfile with;

FROM lambci/lambda:build-python3.6

RUN yum -y install mysql-devel

And serverless.yml with;

custom:
  wsgi:
    app: project.wsgi.application
  pythonRequirements:
    dockerizePip: true
    dockerFile: Dockerfile
    dockerExtraFiles:
      - /usr/lib64/mysql/libmysqlclient.so.18
#      - /usr/lib64/mysql57/libmysqlclient.so.1020

(Tried both extra files, as could not see the commented out file inside my Docker container...)

Is this something to do with the above not installing a high enough version of mysqlclient for Django 2.2?