WIP: docker in docker fix

[sukovanej]

Please note that - I'm not a JS dev, I actually haven't studied the whole project and the fix is something I really needed to make 'it work'. I believe it's not the best solution and I'm totally open to a discussion.

So the problem is that in CI I need to run sls deploy and the CI stage itself is running in a container. After some research, I found that the bindPath is assigned to its host $(pwd)$ value. The problem is, the 'docker container inside the CI docker container' thinks the host is the main CI container so the path is actually wrong. So I simply shared $(pwd) from the original host to the CI container as a host_pwd env variable and let the bindPath to take that value in case the dockerInDocker option is true.

In .gitlab-ci.yml

    - export host_pwd=$CI_PROJECT_DIR

and of course, the whole stage must be run in a container capable of running another container and in config.toml there must be docker socket specified.

and in serverless.yml

custom:
    pythonRequirements:
        dockerInDocker: true
        dockerizePip: true

What do you think?

[dschep]

I think a bind path option would be preferable. And you'd use it like:

custom:
    pythonRequirements:
        dockerizePip: true
        dockerBindPath: ${env:CI_PROJECT_DIR}

@kichik @heri16, this could also be a solution to weird windows bind paths (tho definitely still less user-friendly than the current solution)

[kichik]

Could be. In that case it would be cool if we can also have this as a command line argument. That's easier to do with different development environments.

[dschep]

@kichik any option can be a CLI option using serverless' variable system 😉

custom:
    pythonRequirements:
        dockerizePip: true
        dockerBindPath: ${opt:dockerBindPath}

then use sls --dockerBindPath /foobar package

[heri16]

@sukovanej could you elaborate on what is meant by the bindPath is assigned to its host pwd value? 'docker container inside the CI docker container' thinks the host is the main CI container? You meant the docker container spawned by this plugin? Are you able to check the servicePath returned by the serverless framework inside your CI?

I have a feeling this is a problem caused by an incorrect setup/wiring of Docker, and not likely a problem with the plugin.

[sukovanej]

@dschep Yep, that seems like a better solution to me. 👍

[brettdh]

@sukovanej I have the same issue - using dockerizePip in Gitlab CI, which itself runs docker-in-docker. Oddly, I only noticed a problem starting recently, when I upgraded the plugin from 3.0.7 to 4.2.4. Thanks for tracking this down and figuring it out.

Do you still plan to adjust this PR to implement a general bindPath option, or should I perhaps create a new one to do the same?

@heri16 it's possible that this is just a band-aid. I don't think I fully understand the problem yet, but I have observed the symptom:

Could not open requirements file: [Errno 2] No such file or directory: '/var/task/requirements.txt'

but I also recall other volume-related issues in this CI setup, so I'd need to dig some more to be sure that it's the same issue.

[marlonchalegre]

Is this issue solved? I'm facing the same problem to build in a docker in docker environment

[campellcl]

Yup, same issue here when using Jenkins CI with docker in docker.

[AndrewFarley]

I’m going to help pull this over the finish line. I will try to test this out more completely and add a test for it and recommend it’s merging when complete. I think DID is quite critical for some folks ci systems. I myself use kubernetes quite heavily. I will test and fiddle with this today. Might have a few more commits to this to add some tests and docs. Cheers

[campellcl]

@AndrewFarley I know DinD is not necessarily a recommended approach these days, but I agree; it still is critical in CI/CD systems such as Jenkins. In the case of Jenkins, the official tutorials still support it's usage. So it would be very helpful if serverless-python-requirements functioned out-of-the-box in CI environments that still leverage the legacy DinD approach. Thanks a ton for looking into this!

[AndrewFarley]

So, I hacked on this a bit this weekend and have some preliminary results.

First, the concept from this MR does work, the challenge is how to make it work reliably. Let me discuss/debate/share some thoughts...

So, first assuming we have docker in docker support, means that I should be able to run the docker command from within docker so I can look-up information about the docker container I am in right now. In order to do that I need to know what container ID I am running in, for that I found a neat cgroup trick I didn't know to lookup what container ID you have from within' the container..

With that above, I can query where the docker volume overlay is on the host and then can use this as a prefix for any bind url(s). This will prevent what this MR's author had to do to make this MR work, which is to know the actual storage path of the host and set that as an environment variable. In many cases, you won't know because it won't be hardcoded (eg: Kubernetes), and doing so is an undue burden on users.

So, I worked out a PoC doing exactly this and it seems to work reliably. However... this is one of a multitude of use-cases and configurations of how docker would be setup. Off the top of my head, an edge case that would fail is if any volumes are mounted which would contain the actual code that you plan to serverless deploy from. Then your root overlay would not contain the serverless data, but the volume mount would. I would need to add code to sift through the volumes mounted in the data from docker inspect and detect if the code you were trying to deploy is within' those volumes, and if so, use THAT as a file base instead of the root volume overlay from the trick above. I'm thinking that I may need to add "two" new features, one to try to auto-detect the overlay/volume from the host and make it work magically, the second is to just allow you to specify what the volume will be if you know where the data volume is on the host. The more I look at and think about this the more daunting of a task this becomes because I see all the edge cases and differing setups.

My goal, for any MR/PR I would create or approve for this would be such that...

• Docker in Docker works from a plain ubuntu 18.x instance just running Docker CE
• Docker in Docker works from AWS EKS Kubernetes (default config)
• Docker in Docker works from Google Cloud GKE (default config)

I probably won't have time to fiddle more until next weekend. But will post here if/when I do. Feel free to respond/comment if you have any input/ideas/support to anything I mentioned.

[campellcl]

@AndrewFarley Thanks again for looking into this! I haven't dealt with Kubernetes, but In my opinion I don't think it is too unreasonable for the user to have to know the which volumes they have mounted on the host os. If it makes your life easier, it may be worthwhile bringing the discussion back to CI/CD, and consider the standard Docker configurations for the major CI/CD players.

From the Jenkins standpoint, I believe the user is utilizing bind mounts instead of volumes. On the host OS, there is both a DinD container running, and the Jenkins container running. The Jenkins container deploys one-or-many additional "Agent" containers which reference the mounted volumes containing the actual Serverless code to be deployed. The Serverless deployment is happening inside the Jenkins "Agent" container, and not inside the Jenkins container itself.

This is indeed difficult, because theoretically speaking you could be deploying Serverless code inside an arbitrary number of levels of nested Docker containers. I would just try and support the configurations adopted by the major CI/CD players instead of worrying about all potential DinD use cases. As DinD is slowly being replaced anyway by bind mounting the the Docker socket.

[brettdh]

@ccampell I would disagree that DinD is being replaced by bind-mounting the docker socket; each approach has its advantages and disadvantages. The article you linked is over four years old, and DinD remains a "recommended configuration" for building docker images with Gitlab CI.

[campellcl]

@brettdh Fair enough, that is a valid point! Here is a more recent article discussing the pros and cons of each approach. I'm inclined to agree with you, and amend my previous statement. I meant that for the particular use case of being able to run Docker commands from the confines of a containerized CI/CD system, from what I have seen, there is a general trend toward the transition of bind mounting the Docker socket and utilizing the sidecar pattern. But as always, nothing is strictly black and white, and your right there are advantages and disadvantages to each approach.

[AndrewFarley]

There's no reason to debate what the best/current/more secure practice of accomplishing Docker in Docker because this is out of the scope of this plugin. This plugin will/does not care "how" you made your Docker CLI command work inside of Docker, that's on you and/or your admin to accomplish.

@ccampell To your point, I checked in a few systems that I manage and all of them you don't specify where your volumes/binds are mounted they are generated, and while I understand you do, my experience tells me this is unusual. Additionally, most CI workflows (that I use) just check out the code whilst you are already inside your first Docker container, thus you don't know the mount ahead of time. Having a workflow differently seems... odd to me, but I'm not debating that you can know the mount ahead of time. Anyways, as I explained, I overcame that limitation/issue so it's a moot point.

I personally mostly run Kubernetes clusters for clients, and I heavily lean on GitlabCI personally, with the odd Jenkins setup or two and nowadays some Gitlab Actions as well, but to accomplish CI/CD for the serverless framework with this plugin for my clients I typically use an LambCI image directly as the image I start (typically inside Kubernetes) and then from in there install the Serverless framework, check out the code, and run this plugin which in that environment means I don't need Docker in Docker, I already have a comparable environment to AWS that I'm building in. I've avoided needing to use DinD entirely which is why I never really thought this feature was needed. In a sense, I still don't, but I don't want to generalise and assume.

In my research for solving all of this, I did stumble onto a DinD alternative which I think is very interesting and might handle this whole thing a LOT better, but it requires a lot more time investment than I'm willing to give. I thought this might be interesting to you folks who do Docker in Docker and CI stuff: https://github.com/GoogleContainerTools/kaniko . If that does what it says it does, I personally would probably shift most of my CI automation over from DinD to Kaniko simply for security purposes in any future CI setups of mine. But, alas, imho that may be out of the scope of solving this direct issue, if only because I have no experience with that tool at this time. Again keep y'all posted will have a branch you guys can try soon to get some feedback.

[AndrewFarley]

@ccampell / @brettdh (or anyone) Want to test for me? I tested via Docker via the mounted sock and via the DinD sidecar, both seem completely functional with or without caching. There are no tests or anything, there is verbose logging as well. I created a fork, you can use it with...

npm remove --save serverless-python-requirements
npm install --save github:andrewfarley/serverless-python-requirements#dockerindocker

From here, you either run git add/commit/push and let your CI do its thing depending how you have it setup, or just run serverless package to test out the packaging. If you are in docker, it should detect docker-in-docker and detect the volume path and run properly.

I confirm this functional on...
• OS-X via /var/docker sock volume mount (wow, cool huh?)
• Linux Ubuntu 18.04 Test Server with Docker via .sock volume mount
• Linux Ubuntu 18.04 Test Server with Docker via dind sidecar
• Briefly worked on Kubernetes, but need to test more

Here's a snapshot of it working nicely on my Gitlab AWS autoscaling runners. NOTE: This of course did not work before, it now works beautifully.

Feedback welcome!!!

[AndrewFarley]

I'm going to close this MR I've taken over the work and will finish it in #484. Please continue the conversation and add comments over there.