Fix broken integration test TestRegularWorkspaceTasks

[jenting]

Bug description

The integration test TestRegularWorkspaceTasks is broken with the errors

=== RUN   TestRegularWorkspaceTasks
 === RUN   TestRegularWorkspaceTasks/ws-manager
 === RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks
 === RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/init
     tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
 === RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/before
     tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
 === RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/command
     tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
 --- FAIL: TestRegularWorkspaceTasks (23.75s)
     --- FAIL: TestRegularWorkspaceTasks/ws-manager (23.75s)
         --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks (23.75s)
             --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/init (9.68s)
             --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/before (7.01s)
             --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/command (7.06s)

The analysis root cause could be #8800 (comment)

Steps to reproduce

# Switch to preview environment
./dev/preview/install-k3s-kubeconfig.sh

# Run integration test
cd test
go test -v ./... \
   -kubeconfig=/home/gitpod/.kube/config \
   -namespace=default \
   -run=TestRegularWorkspaceTasks

Tasks

• Fix supervisor test
• Fix agent part which checks existence of files

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[jenting]

In the current code, I encounter the certificate issue, waiting for PR #9553 fixed.
We could revisit this issue once the certificate issue is addressed.

Related Slack Thread.

[jenting]

The supervisor opens the port on 22999 on IPv6 localhost address

$ netstat -ntlp  | grep 22999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::22999                :::*                    LISTEN      -

Port forward to address 0.0.0.0 with local port 32999 to remote port 22999

$ kubectl port-forward --address=0.0.0.0 pod/ws-87285dbc-7400-4afb-85e8-a6de243cfe5c 32999:22999
Forwarding from 127.0.0.1:32999 -> 22999

Access with supervisor endpoint

$ curl -XGET 'http://127.0.0.1:32999/_supervisor/v1/status/tasks'
curl: (52) Empty reply from server

$ k port-forward --address=0.0.0.0 pod/ws-87285dbc-7400-4afb-85e8-a6de243cfe5c 32999:22999
Forwarding from 0.0.0.0:32999 -> 22999
Handling connection for 32999
E0503 09:36:56.344879   45463 portforward.go:406] an error occurred forwarding 32999 -> 22999: error forwarding port 22999 to pod 88503d150548aaf02c1754d273767d2d9c0ffd1ad661a8819218ac91ad236753, uid : failed to execute portforward in network namespace "/var/run/netns/cni-1d91c723-efc9-a8d1-ea28-513326b839ed": failed to connect to localhost:22999 inside namespace "88503d150548aaf02c1754d273767d2d9c0ffd1ad661a8819218ac91ad236753", IPv4: dial tcp4 127.0.0.1:22999: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found 
E0503 09:36:56.345241   45463 portforward.go:234] lost connection to pod

I tried change the port-forward with --address=localhost

$ kubectl port-forward --address=localhost pod/ws-87285dbc-7400-4afb-85e8-a6de243cfe5c 32999:22999
Forwarding from 127.0.0.1:32999 -> 22999
Forwarding from [::1]:32999 -> 22999

The supervisor opens the port on 22999 on IPv6 localhost address, and opened port 32999 on IPv4 and IPv6.

 netstat -tnlp | grep 32999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 10.0.5.2:32999          0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:32999         0.0.0.0:*               LISTEN      45942/kubectl       
tcp6       0      0 ::1:32999               :::*                    LISTEN      45942/kubectl

Access with supervisor endpoint

$ curl -XGET 'http://127.0.0.1:32999/_supervisor/v1/status/tasks'
curl: (52) Empty reply from server
$ curl -XGET 'http://[::1]:32999/_supervisor/v1/status/tasks'
curl: (52) Empty reply from server

$ kubectl port-forward --address=localhost pod/ws-87285dbc-7400-4afb-85e8-a6de243cfe5c 32999:22999
Forwarding from 127.0.0.1:32999 -> 22999
Forwarding from [::1]:32999 -> 22999
Handling connection for 32999
E0503 09:42:26.602006   46159 portforward.go:406] an error occurred forwarding 32999 -> 22999: error forwarding port 22999 to pod 88503d150548aaf02c1754d273767d2d9c0ffd1ad661a8819218ac91ad236753, uid : failed to execute portforward in network namespace "/var/run/netns/cni-1d91c723-efc9-a8d1-ea28-513326b839ed": failed to connect to localhost:22999 inside namespace "88503d150548aaf02c1754d273767d2d9c0ffd1ad661a8819218ac91ad236753", IPv4: dial tcp4 127.0.0.1:22999: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found 
E0503 09:42:26.602432   46159 portforward.go:234] lost connection to pod

@gitpod-io/engineering-workspace Any thought or comment?

[princerachit]

Hi, I am looking into this error. Will update this issue once I find something.

[princerachit]

Workspace logs

The workspace created through this test has the following logs. Notice last line not connected to Gitpod server error.

...
...
...
Web UI available at http://localhost:23000/
[15:06:26] Extension host agent started.
{"ide":"IDE","level":"info","message":"IDE readiness took 5.307 seconds","serviceContext":{"service":"supervisor","version":"commit-56fe9e79016d84f0e5acff880b6b1fc18e3e9707"},"severity":"INFO","time":"2022-05-04T15:06:26Z"}
{"ide":"IDE","level":"info","message":"IDE is ready","serviceContext":{"service":"supervisor","version":"commit-56fe9e79016d84f0e5acff880b6b1fc18e3e9707"},"severity":"INFO","time":"2022-05-04T15:06:26Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"not connected to Gitpod server","level":"error","message":"error tracking supervisor_readiness","serviceContext":{"service":"supervisor","version":"commit-56fe9e79016d84f0e5acff880b6b1fc18e3e9707"},"severity":"ERROR","time":"2022-05-04T15:06:26Z"}

[princerachit]

The port-forward will always fail for supervisor port. When kubectl tries to port forward it does that in context of a network namespace. This network namespace is the one which was originally created for the pod. This is the same nw namespace where ./workspacekit ring0 command was run (the entrypoint). workspacekit creates a new set of namespaces in ring1 which in turn creates another set of namespaces in ring2. This ultimate set of namespaces is wherethe supervisor is run see architecture.

Therefore, kubectl does not have access to the namespace where supervisor is running, hence the error.

Proposed solution

1. Modify the api.Supervisor method to port forward from correct nw ns (PREFERABLE); OR
2. Find another way to check the status of task

@jenting

correct me if I am wrong @gitpod-io/engineering-workspace

[jenting]

The port-forward will always fail for supervisor port. When kubectl tries to port forward it does that in context of a network namespace. This network namespace is the one which was originally created for the pod. This is the same nw namespace where ./workspacekit ring0 command was run (the entrypoint). workspacekit creates a new set of namespaces in ring1 which in turn creates another set of namespaces in ring2. This ultimate set of namespaces is wherethe supervisor is run see architecture.

Therefore, kubectl does not have access to the namespace where supervisor is running, hence the error.

Proposed solution

1. Modify the api.Supervisor method to port forward from correct nw ns (PREFERABLE); OR
2. Find another way to check the status of task

@jenting

correct me if I am wrong @gitpod-io/engineering-workspace

Thanks Prince's analysys.
I'd prefer solution 1, but how do we get the supervisor original network namespace?

[princerachit]

kubectl cli does not have any configuration option for port-forward to choose the namespace (container namespace not kubernetes namespace) in which the port forwarding should run. Neither the kubelet code takes such configuration parameter.

One way to solve this could be run an exec command through kubectl inorder to expose the port. I still need to figure out how to do this and if this is feasible.

Other way is to make use of integration test agent to enter relevant namespace and then run a curl on the supervisor endpoint.

[csweichel]

Could we not get the owner token of the workspace and talk to the supervisor API using the IDE URL?