Feature Request: Support/Implement Layers (new AWS Lambda Feature)

[AndrewFarley]

Hey guys, I wanted to start a bit of a discussion on this topic. I haven't had time to look at or try it, but after the recent AWS announcements it got me thinking this might really help a lot of people to build/deploy serverless apps, specifically ones that need to use python deps included via this plugin (as all of mine do). Right now, the "cost-to-entry" is higher in that many devs have to have docker installed just to use this plugin, and on higher security/locked down environments you can't even have docker.

So the proposal/idea, this layers feature would allow a "ops" guy or even a CI/CD system to push "layers" that add just your python requirements into lambda, and thus making the actual code we need to push on every deploy very thin. At the moment some of my projects I'm pushing upwards of 20MB of compressed code 98% of which is requirements. How amazing would it be to not wait minutes for that to upload and process on Lambda? Instead we would be pushing 200Kb project-specific code that doesn't need to use this plugin whatsoever. This would allow for serverless to be used in higher security environments, and would enable developers to move/deploy/iterate faster.

At first thought, possibly nothing would need to be added to this plugin. Might just be someone would need to use this plugin in the "requirements" layer (aka, a different serverless stack/project). And then the devs can just easily include that layer in their serverless.yml on their project without having to use this plugin, or docker at all. It's a really neat idea, I haven't tried any of it yet. Was curious if anyone had, or anyone had other thoughts/ideas. If anyone tries layers or a comparable pipeline to what I described above, I'd love to hear your experiences. If/when I get some time to evaluate this in the coming weeks I'll share my experiences also.

See:
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
https://serverless.com/blog/publish-aws-lambda-layers-serverless-framework/

[dschep]

I implemented the framework's support for layers and definitely want to add support for layers to this plugin. I'm not working on that now so feel free to take a stab at it. Here's some links with docs

• https://serverless.com/blog/publish-aws-lambda-layers-serverless-framework
• https://serverless.com/framework/docs/providers/aws/guide/layers/

[str3tch]

Well I had the same idea. I recently manually put the python requirements into a layer - reduced my function zip uploads from 8MB+ to now ~30kb or so. So much better!

The more I think about it, this is my approach, which I am slowly starting to implement./

• Use Lambda, not docker.
• POST the packages you want to install, along with a name for the Layer
• In the Lambda, use the pip to install the packages to the local filesystem
• Create/update the Layer
• return the ARN of the Layer
• Use the Layer in the Serverless.yml

Now there's no need for Docker! Nice and easy. People can host this as a central service, as @AndrewFarley suggested, or just sls invoke the function with the json payload (packages + layer name etc)

If anyone can think whether this can be put as a plugin, let's talk..!

[dschep]

That's neat @str3tch! I'm happy to chat about doing that as a plugin (a new one, not this one). If you and @AndrewFarley wanna join me to chat about this, I suggest you all join the serverless slack and I'll make a channel for discussing this stuff 😄

Also, check out these notes I'll be pushing to my blog post soon. The cf exports stuff might be of interest if you're keeping your layer in a different serverless project(probably want to, so that you don't deploy it every time you deploy the functions) https://github.com/serverless/blog/blob/70b488b42682783696e31f131f7aafa506be230c/posts/2018-11-29-publish-aws-lambda-layers-serverless-framework.md#some-tips-on-working-with-layers

I've been toying with a similar idea (scheduled lambda that updates a layer) for my sample geoip layer since that data changes constantly: https://github.com/dschep/geoip-lambda-layer

[AndrewFarley]

I am not sure it completely warrants a different plugin, since this plugin largely does what we need might just need some examples in the examples folder of how to leverage this plugin with multi stacks leveraging a dedicated "layers" stack(s). But I'd love to hash the ideas out! I'll jump on the serverless slack now. Invite me to a channel... @farley on Serverless Slack.

Cheers!

[AndrewFarley]

Also... just throwing it out there, but this also might make a wonderful fully managed service (which you could install as a serverless plugin) that you uploaded your requirements.txt to and it generated the layer and pushed it to your AWS account. With assume role and very strict permissions this would be amazing, and really lighten the load for deploying Python-based projects from any OS. It's going the opposite direction as far as client-side complexity required to deploy, pushing the complexity to the server/service to generate. If we don't want to host this kind of service ourself, it could also be run in AWS via Fargate or something, build right on AWS and then push to a layer, and the plugin could manage exactly that... Super neat idea! :P Maybe this is more in line with the other plugin idea?

[dschep]

@str3tch, how do you deal with lambda's env not having build dependencies? or does it only work for packages that provide manylinux wheels?

[bsamuel-ui]

I was reading that as running a service that does the builds, hence the need for ECS.

I think the hosting problem is that this comes with all the scariness of running Other People's Code on your service, besides how you fund it if it gets popular.

That said, it may be very easy to self-host this. We've been running an API that at work that is runs jobs on ECS (it's compatible with both classic ECS and Fargate) and solves issues with failed jobs and such, so I'd just have to get permission to open source it, and then take the time to factor it out into a standalone delpoyment package. But, while we have to do interesting stuff with auto-scaling groups and lifecycle hooks, it's all basically a cloudformation template.

[AndrewFarley]

@bsamuel-ui I had to chuckle... isn't every NPM or Python package ever also "Other People's Code" when it really comes down to it? :P

[bsamuel-ui]

It absolutely is, though, here you're promising to run arbitrary packages for free, and I suspect you'll wind up either putting stringent caps on build times or playing whack-a-mole with bitcoin miners.

[str3tch]

I had a play with this just before Christmas.. only just remembered.

The issue is that in a real lambda, there is no pip, or not one that I know how to call. So you have to include it with your deployment package and add it to the python path.

wheel packages install fine, but tar.gz ones fail because I can't seem to get setuptools on the path, when it spawns a task to call python setup.py egg.info. See the readme in my repo below.

https://github.com/str3tch/python-lambda-layer

(warning! rough and ready code, just trying to POC the idea)

If anyone has any ideas of how to do this, let me know!

[david-mkl]

I've got a POC for this in #310 that I'd love to get feedback on.

I haven't tried to solve the problem of installing the requirements in a Lambda function. So it will still require docker. This will simply just package the requirements separately and create a layer from that artifact.

[luisdemarchi]

Hi.

I think it would be perfect and it makes a lot of sense for this plugin to support layers. Mainly keep the current settings and with some extra command pass the libraries of each module to layers.

[dschep]

@luisdemarchi I just merged @david-mk-lawrence, please take it for a spin by installing this plugin from github: npm i unitedincome/serverless-python-requirements

I still need to test it more thoroughly myself, but baring that, I'll cut a release soon 😄