[WIP] Docker in Docker support #484
Conversation
|
Pinging for visibility from #198 - Looking for reviewers/testers... |
|
@AndrewFarley I'll have a chance to run it through our CI/CD Sunday to test it out. Thanks again! Edit (3/8/2020): + cd JobManagementService
+ npm install
+ sls deploy --stage jenkins -v
Serverless: Generated requirements from /var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/requirements.txt in /var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/.serverless/requirements.txt...
Serverless: Installing requirements from .cache/serverless-python-requirements/717d2b3fe394a147791937181371c1a35c16a04363f8a54e9abc8da17b26380f_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.8
Serverless: Docker-In-Docker: servicePath: .cache/serverless-python-requirements/717d2b3fe394a147791937181371c1a35c16a04363f8a54e9abc8da17b26380f_slspyc
Serverless: Docker-In-Docker: We have detected an docker-in-docker configuration. NOTE: This feature is in beta for this plugin, verbose output for now
Serverless: Docker-In-Docker: Detected container: 91cad0f14d0a061338c37a664f58fd10f2d6336d9c32f6ca9a1113ec192cb2d6
Error --------------------------------------------------
Error: docker not found! Please install it.
at dockerCommand (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/serverless-python-requirements/lib/docker.js:45:13)
at getBindPath (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/serverless-python-requirements/lib/docker.js:153:14)
at installRequirements (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/serverless-python-requirements/lib/pip.js:198:37)
at installRequirementsIfNeeded (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/serverless-python-requirements/lib/pip.js:555:3)
at ServerlessPythonRequirements.installAllRequirements (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/serverless-python-requirements/lib/pip.js:634:29)
at ServerlessPythonRequirements.tryCatcher (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromise0 (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/promise.js:729:18)
at _drainQueueStep (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (/var/jenkins_home/workspace/EP_test-docker-in-docker-support/JobManagementService/node_modules/bluebird/js/release/async.js:15:14)
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)I noticed that the cached container id didn't match up. So I added the following additional lines to this Service's And I re-ran it again, and received a different error (which may not be related to your changes): Note: Non-truncated logs available here @AndrewFarley Any other logs I can produce to help with this? |
|
@AndrewFarley By design the Jenkins Agents are running as non-root users, which may explain the In regards to the cache, you may wish to add a check to ensure that the cache matches up with the currently deployed Docker container instance, and forcefully invalidate the caches if they do not match. Then again, this may not be an issue if running from clean install with no previously cached files. |
|
@ccampbell if it says docker command not found then you do not have docker in docker support or the image you are using doesn’t have the docker command installed which is a prerequisite for this plugin to work at all with docker or dind support. Are you sure you need docker in docker support? It sounds to me like you can make the Jenkins runner use whatever image you want, correct? If so you can have it run the requisite lambci image and disable the unnecessary docker support. |
@AndrewFarley I know that is what the error usually means, but in this context docker is installed; just not in the Agent container. How else could you detect the docker container the process was running in? Your output here: To my knowledge, you wouldn't be able to pull the container Id (via the method you previously linked in #198 ) without Docker installed.
The Jenkins CI/CD setup with DinD is a bit convoluted. I have created an Ubuntu 18.04 Docker image with both NodeJS and Python installed and configured. The Jenkins tutorials instruct the user to run the DinD image, and run the
The Ubuntu 18.04 image I created serves to mirror the development environment on my local machine for testing. The Of course there are alternatives to this, such as not running Dockerized Jenkins, bind-mounting the Docker socket and using the sidecar pattern, or creating your own container with Docker installed and then installing and configuring Jenkins within the Dockerfile somehow. So do we really need docker-in-docker? No, but the Jenkins CI/CD folks recommend the Dockerized Jenkins approach for getting up and running quickly. So it's a pretty common use case to have a Jenkins container deploying another Agent container, which would be executing the actual Serverless deployment commands. I can try installing Docker in the Ubuntu 18.04 Agent container, but this is redundant. The Agent doesn't need to run Docker commands, however the parent Jenkins container does. Does that clarify our intended use case? |
|
Detecting the container id doesn’t relate to the ability to run docker or the docker command. It honestly sounds like all you need to do is install docker in your pipeline before running serverless if you are sure it is bind mounted to make dind work. Could you try installing docker? Or use a different image with it preinstalled |
You're right, sorry; Docker isn't installed in the Ubuntu 18.04/Agent container. I should have made that a bit more explicit. But my point is that it really doesn't need to be. The Jenkins Agent container doesn't need to execute Docker commands, only the Jenkins If it would be helpful for you, I can try installing Docker on the Ubuntu 18.04/Agent container. But again, we wouldn't be utilizing Docker commands inside the Agent container, so this would be a step we would only undergo if it made the plugin function correctly. In some use cases, maybe there would be a need to run Docker commands inside the child Agent container, but I don't think that is nearly as common of a use case. We want Docker-in-Docker support in the sense that the plugin is supported by the child Agent container that the Jenkins container deploys (via the DinD image). Does that make sense? @AndrewFarley Would you like me to try again with Docker installed in the Agent container to aid in debugging? Also, I'm not sure it will work without invalidating the caches. Which appears to cause that second error which I do not have a solution for. |
|
I'm not sure I entirely understand. It sounds like you "think" you want docker-in-docker support but you don't really and should just use a lambci image as the jenkins "agent" runner. Or, if your setup is configured so DinD should work, you don't have docker installed to confirm if it actually works. I would first go install docker in your agent container and see if it works. If it does, then re-try this fork/branch. If it doesn't see my next comment below. If that doesn't work, let's talk interactively to figure this out. @ccampell Can you jump on the serverless slack team by visiting this url: https://serverless.com/slack and chat with me interactively... |
|
@ccampell If your parent container supports running commands to initialise Jenkins runners, then it sounds like a simple solution would be to make a runner that is meant for serverless that uses the lambci images, on which you install a jenkins runner as well. Then you can just start an agent that already has an environment which wouldn't require docker within it. Instead of using a generic |
|
@AndrewFarley Yeah, It's not the most graceful setup; but it's what the Jenkins folks recommended. We have DinD running per the jenkins tutorials: docker container run --name jenkins-docker --rm --detach \
--privileged --network jenkins --network-alias docker \
--env DOCKER_TLS_CERTDIR=/certs \
--volume jenkins-docker-certs:/certs/client \
--volume jenkins-data:/var/jenkins_home \
--volume "$HOME":/home docker:dindAnd we have the parent Jenkins container running: docker container run --name jenkins-tutorial --rm --detach \
--network jenkins --env DOCKER_HOST=tcp://docker:2376 \
--env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 \
--volume jenkins-data:/var/jenkins_home \
--volume jenkins-docker-certs:/certs/client:ro \
--volume "$HOME":/home \
--publish 8080:8080 jenkinsci/blueoceanThen for our Jenkins pipeline ( pipeline {
agent {
dockerfile {
filename 'Dockerfile'
}
}
...
stages {
stage('Deploy some example Serverless service') {
steps {
sh '''
cd someDir
npm install
sls deploy --stage jenkins -v
'''
}
}
stage('Test the deployed Serverless service') {
steps {
sh '''
pytest --verbose --junit-xml ./test-reports/results.xml test/IntegrationTests/TestService.py
'''
}
}
}
}And to be complete, here is some of what our Dockerfile looks like: FROM ubuntu:18.04
...
RUN apt-get install -y nodejs
...
RUN npm install -g serverless
...We don't have Docker installed in the child "agent" container that the parent Jenkins container spawns (i.e. the
It is the above @AndrewFarley As per your suggestion, that is a good idea and would simplify our CI/CD setup. However, we don't really want to "very closely mimic the live AWS Lambda environment " as in lambdaci. Our CI/CD pipeline does a full live deployment via our agent container, and we run tests in the live cloud. We have had discrepancies simulating the cloud locally via plugins. So we prefer to test against a live deployment under a unique stage name. I haven't used lambdaci before so I can't speak to its ability to replicate the cloud offline. I'll hop on the serverless chat and we can discuss tomorrow, if you are available? I wish I could make this more clear, sorry for being so verbose! Again, thank you for your time on this! The plugin works great when we deploy from the host machine. It does not work at all when we have Jenkins run the deployment command from inside the "agent" container as I have demonstrated above. |
|
@ccampell I think I am getting a better picture of what you have in place, but still not a full picture. I think you have the wrong picture of the "mimicking the Lambda environment" means. In that, in this scenario all it means is you are using a different container. From what I see above, it sounds like you could just change your Dockerfile to If you made this plugin work with Docker support enabled, that's all it is doing for you, starting up another docker container with this exact image. You would just be simplifying things, removing the unnecessary dependancy of starting yet another docker container for no reason, is all I'm saying. Finally, I must say, if you won't budge on any of the above suggested items I've mentioned, then I'm not sure I can help you get this plugin working for you. You have to make your environment work with DinD properly before this plugin can work, or you need to have this plugin working in a comparable environment to Lambda when it runs so it can generate compatible requirements. This is "your" problem to solve, not magically this software. Anyways, Slack me up when you get a chance, I'm in a weird timezone in the future (Auckland, NZ) but ping me on that serverless slack and we can find a time that we're both online see if I can get a better understanding. |
|
@AndrewFarley I see what you're saying! I thought you were advocating to use
I misunderstood, and thought you were pitching this as a workaround for not being able to I did not realize that there were additional dependencies for the plugin besides those required for the Serverless framework itself, and those brought in by npm during the plugin's installation. I'm not trying to be obstinate, I will definitely try your suggestion of using the I don't expect the software to magically solve any problems. I am aware that it is my problem to get the software working. I had just hoped that it would still function in a child Docker container without additional modifications, like it does on the host OS. I am definitely guilty of (perhaps incorrectly) thinking that #484 would solve this problem. I have spent valuable time explaining my organizations' use case above, in order to draw attention to a fairly common CI/CD setup that is not currently supported out-of-the-box by the plugin. I did not go through this trouble, with the expectation that you would solve this problem for us. As I have tried to reiterate above; I appreciate you spending your valuable time both: responding, and considering our use case as it pertains to #484 if/where applicable. I am working from the United States' Eastern Coast (EST time). I'll try and ping you on slack at a time that is reasonable for Auckland, NZ. Thank you again, for all your efforts on this! |
|
Used the temporary fix with VSCode's dev in container, running serverless cli. Was getting the /var/task/requirements.txt not found error, after fix installed, it worked fine, deployed to aws Ok. |
|
This fails for me on GitLab CI with: |
|
I've tried setting The logged docker run command is: I'm using |
|
The only thing that's worked for me was reverting back to the 5.1.0 release and setting It seems that this PR mainly attempts support the bind mounting /var/run/docker.sock approach rather than docker-in-docker. As with docker-in-docker the volume mounts actually work (as far as I could tell). UPDATE: |
|
Just tested this branch locally and it worked nicely. I would love to be able to do the following:
Currently i have a bloated Whats left in order to get this PR moving again? |
|
Please see #550 |
|
Since it's my baby, I'll help this through the finish line. I'm glad this has helped some people, I confirm it helps make this plugin work for me in some previously challenging environments. I think what's left really is on the checklist above... Rebase |
|
Any updates on this @AndrewFarley? Unfortunately gitlab dind still doesn't work for me on your branch. |
|
@alecgerona I'm not sure exactly what's missing or needed for this branch to be able to merge, its usefulness seems questionable at best and it doesn't seem to have solved an actual problem. At arms length when I heard of this desire/interest and saw the original MR I thought "yeah, I have had this problem, so let me implement this feature". And I did, but, especially having gone through it, thinking of its actual uses, every alternate use of this plugin is better and in (some) ways simpler than using DinD. As I said above I believe, I've used this branch successfully with Gitlab (self-hosted) dind runners on Kubernetes, and locally on Linux and Mac without issue. Although, that cluster with Gitlab I ended up removing DinD support because it's insecure as hell. Generally in my experience anyone doing DinD is often a sign of "doing something wrong" and there's often a better solution, as my thread above with someone highlighted. There seems to be a big lack-of-understanding of how "else" this plugin can be used besides DinD. If you're already "in" docker, you can just be sure to run in a Linux container with Python support so it can already work without requiring DinD. If you need to run this along side whatever native tools you have that are in a docker container, just put this in your docker container and add NodeJS and then run your serverless commands in that container. You just bypassed the need for DinD and the bad/insecure practice of it. I'd love to hear if there's anyone subscribed to this thread that can really justify continuing work on this, otherwise I'll likely just close and abandon the PR. Almost every use-case I've heard can be done differently, and better from a security point of view. Thoughts? |
|
Hey @AndrewFarley - it's been a long time since this PR was proposed. I'm going to close it, if you feel like the issue is valid, please open a new issue or a new PR against the latest main branch. Thanks 🙇 |
DO NOT MERGE, STILL WORK IN PROGRESS
Replacing/deprecating: #198 from @sukovanej
Purpose
When running Docker-In-Docker this plugin would fail to run because the volume mount path given is based in the docker container, not the host.
Reasoning
To help increase the amount of compatible CI/CD systems/configurations that this plugin can run on. Namely Gitlab Autoscaling Runners and Jenkins Runners.
Result
To implement this I did the following...
/.dockerenvfiledockerizePipto true, then we are indocker-in-dockermodeAppdirplugin for caching because this may produce a volume mount that is invalid. This edge-case happened on two of the systems I manage, so I came up with a solution... instead it puts a folder at the root of the volume to use for caching purposes. Yeah I know... another folder in a random place, but I don't have a reliable alternative.dockerInDockerPaththat allows the user to specify what volume path to pass into docker. This will prevent the system from trying to look this data up. According to a few authors in WIP: docker in docker fix #198 they said they have "fixed/known" mount points. And although I believe the automatic detection in this plugin will handle it, incase it doesn't this is a fallback.Related to:
#198
Instructions:
To try this PR, from the root of your project run the following commands. Assumably run these from within' your first layer of Docker, in a container with a functioning docker command (eg: run
docker pssuccessfully). Also, ensure your serverless.yaml hasdockerizePip: true. It shouldn't matter if your Docker command works because of volume mounting the docker.sock, or if you are using the sidecar mechanism that ends up hosting docker on a TCP port routed through the sidecar. I confirmed both situations work.TODO: