Skip to content

Commit 18c9ca4

Browse files
greboisgrebois
authored
Adding documentation on how to configure Kops to use registry-proxy (#64)
1 parent 6dbce2f commit 18c9ca4

File tree

2 files changed

+240
-0
lines changed

2 files changed

+240
-0
lines changed

Diff for: ‎docs/kops/README.md

+159
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# How to use docker-registry-proxy with kops
2+
3+
## Install docker-registry-proxy
4+
5+
For running docker-registry-proxy with kops you will need to run it outside the cluster you want to configure, you can either use and EC2 instance and run:
6+
7+
```bash
8+
docker run --rm --name docker_registry_proxy -it \
9+
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
10+
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
11+
-v $(pwd)/docker_mirror_certs:/ca \
12+
rpardini/docker-registry-proxy:0.6.0
13+
```
14+
15+
or you can run it from another cluster, maybe a management/observability one with provided yaml, in this case, you will need to change the following lines:
16+
17+
```
18+
annotations:
19+
external-dns.alpha.kubernetes.io/hostname: docker-registry-proxy.<your_domain>
20+
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
21+
```
22+
23+
with the correct domain name, so then you can reference the proxy as `http://docker-registry-proxy.<your_domain>:3128`
24+
25+
## Test the connection to the proxy
26+
27+
A simple curl should return:
28+
29+
```
30+
❯ curl docker-registry-proxy.<your_domain>:3128
31+
docker-registry-proxy: The docker caching proxy is working!%
32+
```
33+
34+
## Configure kops to use the proxy
35+
36+
Kops has the option to configure a cluster wide proxy, as explained [here](https://github.com/kubernetes/kops/blob/master/docs/http_proxy.md) but this wont work, as nodeup will fail to download the images, what you need is to use `additionalUserData`, which is part of the instance groups configuration.
37+
38+
So consider a node configuration like this one:
39+
40+
```
41+
apiVersion: kops.k8s.io/v1alpha2
42+
kind: InstanceGroup
43+
metadata:
44+
labels:
45+
kops.k8s.io/cluster: spot.k8s.local
46+
name: spotgroup
47+
spec:
48+
image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200528
49+
machineType: c3.xlarge
50+
maxSize: 15
51+
minSize: 2
52+
mixedInstancesPolicy:
53+
instances:
54+
- c3.xlarge
55+
- c4.xlarge
56+
- c5.xlarge
57+
- c5a.xlarge
58+
onDemandAboveBase: 0
59+
onDemandBase: 0
60+
spotAllocationStrategy: capacity-optimized
61+
nodeLabels:
62+
kops.k8s.io/instancegroup: spotgroup
63+
role: Node
64+
subnets:
65+
- us-east-1a
66+
- us-east-1b
67+
- us-east-1c
68+
```
69+
70+
you will need to add the following:
71+
72+
```
73+
additionalUserData:
74+
- name: docker-registry-proxy.sh
75+
type: text/x-shellscript
76+
content: |
77+
#!/bin/sh
78+
79+
# Add environment vars pointing Docker to use the proxy
80+
# https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
81+
82+
mkdir -p /etc/systemd/system/docker.service.d
83+
cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
84+
[Service]
85+
Environment="HTTP_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
86+
Environment="HTTPS_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
87+
EOD
88+
89+
# Get the CA certificate from the proxy and make it a trusted root.
90+
curl http://docker-registry-proxy.<your_domain>:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
91+
echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
92+
update-ca-certificates --fresh
93+
94+
# Reload systemd
95+
systemctl daemon-reload
96+
97+
# Restart dockerd
98+
systemctl restart docker.service
99+
```
100+
101+
so the final InstanceGroup will look like this:
102+
103+
```
104+
apiVersion: kops.k8s.io/v1alpha2
105+
kind: InstanceGroup
106+
metadata:
107+
labels:
108+
kops.k8s.io/cluster: spot.k8s.local
109+
name: spotgroup
110+
spec:
111+
additionalUserData:
112+
- name: docker-registry-proxy.sh
113+
type: text/x-shellscript
114+
content: |
115+
#!/bin/sh
116+
117+
# Add environment vars pointing Docker to use the proxy
118+
# https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
119+
120+
mkdir -p /etc/systemd/system/docker.service.d
121+
cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
122+
[Service]
123+
Environment="HTTP_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
124+
Environment="HTTPS_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
125+
EOD
126+
127+
# Get the CA certificate from the proxy and make it a trusted root.
128+
curl http://docker-registry-proxy.<your_domain>:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
129+
echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
130+
update-ca-certificates --fresh
131+
132+
# Reload systemd
133+
systemctl daemon-reload
134+
135+
# Restart dockerd
136+
systemctl restart docker.service
137+
image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200528
138+
machineType: c3.xlarge
139+
maxSize: 15
140+
minSize: 2
141+
mixedInstancesPolicy:
142+
instances:
143+
- c3.xlarge
144+
- c4.xlarge
145+
- c5.xlarge
146+
- c5a.xlarge
147+
onDemandAboveBase: 0
148+
onDemandBase: 0
149+
spotAllocationStrategy: capacity-optimized
150+
nodeLabels:
151+
kops.k8s.io/instancegroup: spotgroup
152+
role: Node
153+
subnets:
154+
- us-east-1a
155+
- us-east-1b
156+
- us-east-1c
157+
```
158+
159+
Now all you need is to upgrade your cluster and do a rolling-update of the nodes, all images will be cached from now on.

Diff for: ‎docs/kops/docker-registry-proxy.yaml

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: StatefulSet
4+
metadata:
5+
name: docker-registry-proxy
6+
namespace: registry-mirrors
7+
labels:
8+
app.kubernetes.io/name: docker-registry-proxy
9+
spec:
10+
serviceName: docker-registry
11+
selector:
12+
matchLabels:
13+
app.kubernetes.io/name: docker-registry-proxy
14+
template:
15+
metadata:
16+
labels:
17+
app.kubernetes.io/name: docker-registry-proxy
18+
spec:
19+
serviceAccountName: default
20+
containers:
21+
- name: docker-registry-proxy
22+
image: ghcr.io/rpardini/docker-registry-proxy:0.6.0
23+
imagePullPolicy: IfNotPresent
24+
env:
25+
- name: ENABLE_MANIFEST_CACHE
26+
value: "true"
27+
- name: REGISTRIES
28+
value: "k8s.gcr.io gcr.io quay.io us.gcr.io"
29+
ports:
30+
- name: http
31+
containerPort: 3128
32+
protocol: TCP
33+
livenessProbe:
34+
httpGet:
35+
path: /
36+
port: http
37+
readinessProbe:
38+
httpGet:
39+
path: /
40+
port: http
41+
volumeMounts:
42+
- name: ca
43+
mountPath: /ca
44+
- name: docker-registry-cache
45+
mountPath: /docker_mirror_cache
46+
resources: {}
47+
volumeClaimTemplates:
48+
- metadata:
49+
name: ca
50+
spec:
51+
accessModes: ["ReadWriteOnce"]
52+
resources:
53+
requests:
54+
storage: 1Gi
55+
- metadata:
56+
name: docker-registry-cache
57+
spec:
58+
accessModes: ["ReadWriteOnce"]
59+
resources:
60+
requests:
61+
storage: 100Gi
62+
---
63+
apiVersion: v1
64+
kind: Service
65+
metadata:
66+
name: docker-registry-proxy
67+
namespace: registry-mirrors
68+
labels:
69+
app.kubernetes.io/name: docker-registry-proxy
70+
annotations:
71+
external-dns.alpha.kubernetes.io/hostname: docker-registry-proxy.<your_domain>
72+
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
73+
spec:
74+
type: LoadBalancer
75+
ports:
76+
- port: 3128
77+
targetPort: http
78+
protocol: TCP
79+
name: http
80+
selector:
81+
app.kubernetes.io/name: docker-registry-proxy

0 commit comments

Comments
 (0)