Chaos experiments are not failing

[arun-gupta]

Running the Chaos experiment on an EKS 1.9.x cluster shows the following output:

310-chaos-engineering $ chaos run experiments/experiment.json 
[2018-04-24 18:34:38 INFO] Validating the experiment's syntax
[2018-04-24 18:34:39 INFO] Experiment looks valid
[2018-04-24 18:34:39 INFO] Running experiment: Terminating the greeting service should not impact users
[2018-04-24 18:34:39 INFO] Steady state hypothesis: Services are all available and healthy
[2018-04-24 18:34:39 INFO] Probe: application-should-be-alive-and-healthy
[2018-04-24 18:34:39 INFO] Probe: application-must-respond-normally
[2018-04-24 18:34:40 INFO] Steady state hypothesis is met!
[2018-04-24 18:34:40 INFO] Action: terminate-greeting-service
[2018-04-24 18:34:40 INFO] Probe: fetch-application-logs
[2018-04-24 18:34:41 INFO] Steady state hypothesis: Services are all available and healthy
[2018-04-24 18:34:41 INFO] Probe: application-should-be-alive-and-healthy
[2018-04-24 18:34:41 INFO] Probe: application-must-respond-normally
[2018-04-24 18:34:42 INFO] Steady state hypothesis is met!
[2018-04-24 18:34:42 INFO] Let's rollback...
[2018-04-24 18:34:42 INFO] No declared rollbacks, let's move on.
[2018-04-24 18:34:42 INFO] Experiment ended with status: completed

Here is the configuration:

Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-eks.1", GitCommit:"e6c42d312ce5a80461ce10e6543e8fe0346bf065", GitTreeState:"clean", BuildDate:"2018-02-16T21:41:28Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-eks.1", GitCommit:"e6c42d312ce5a80461ce10e6543e8fe0346bf065", GitTreeState:"clean", BuildDate:"2018-02-13T22:26:33Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

This shows that all the services are healthy which is not the expected output.

The behavior is correct on a kops-generated cluster. Here is the output:

310-chaos-engineering $ chaos run experiments/experiment.json 
[2018-04-24 18:43:56 INFO] Validating the experiment's syntax
[2018-04-24 18:43:56 ERROR] Configuration makes reference to an environment key that does not exist: WEBAPP_URL
310-chaos-engineering $ export WEBAPP_URL="http://$(kubectl get svc/webapp-service -o jsonpath={.status.loadBalancer.ingress[0].hostname})/"
310-chaos-engineering $ chaos run experiments/experiment.json 
[2018-04-24 18:44:40 INFO] Validating the experiment's syntax
[2018-04-24 18:44:40 INFO] Experiment looks valid
[2018-04-24 18:44:40 INFO] Running experiment: Terminating the greeting service should not impact users
[2018-04-24 18:44:40 INFO] Steady state hypothesis: Services are all available and healthy
[2018-04-24 18:44:40 INFO] Probe: application-should-be-alive-and-healthy
[2018-04-24 18:44:40 INFO] Probe: application-must-respond-normally
[2018-04-24 18:44:41 INFO] Steady state hypothesis is met!
[2018-04-24 18:44:41 INFO] Action: terminate-greeting-service
[2018-04-24 18:44:42 INFO] Probe: fetch-application-logs
[2018-04-24 18:44:42 INFO] Steady state hypothesis: Services are all available and healthy
[2018-04-24 18:44:42 INFO] Probe: application-should-be-alive-and-healthy
[2018-04-24 18:44:43 INFO] Probe: application-must-respond-normally
[2018-04-24 18:44:46 ERROR]   => failed: activity took too long to complete
[2018-04-24 18:44:46 CRITICAL] Steady state probe 'application-must-respond-normally' is not in the given tolerance so failing this experiment
[2018-04-24 18:44:46 INFO] Let's rollback...
[2018-04-24 18:44:46 INFO] No declared rollbacks, let's move on.
[2018-04-24 18:44:46 INFO] Experiment ended with status: failed

Here is the configuration information:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T20:49:26Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

FYI @russmiles

[arun-gupta]

Need to include EKS kubectl in the PATH and also set KUBECONFIG. Now getting:

[2018-04-24 19:07:41 INFO] Probe: application-should-be-alive-and-healthy
[2018-04-24 19:07:41 ERROR]   => failed: kubernetes.client.rest.ApiException: (403)
    Reason: Forbidden
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 25 Apr 2018 02:07:42 GMT', 'Content-Length': '230'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
[2018-04-24 19:07:41 CRITICAL] Steady state probe 'application-should-be-alive-and-healthy' is not in the given tolerance so failing this experiment

This is even when pods are directly accessible:

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
greeter-rs-4n8rn           1/1       Running   0          6m
name-rs-vf99q              1/1       Running   0          6m
webapp-6f4546695d-rjwwd    1/1       Running   0          7h
webapp-rs-k2jhw            1/1       Running   0          6m
wildfly-594b7f7d97-j2t7b   1/1       Running   0          1d

[Lawouach]

Hi Arun,

Sorry to hear you're hitting a bump.

I think the problem is that your service account probably has not the right permissions to perform the List Pods operation. I assume your EKS cluster has RBAC enabled. You probably want to grant more permissions to your service account.

"pods is forbidden: User "system:anonymous" cannot list pods in the namespace "default""

Why does it use the system:anonymous user? I'm not sure.

Also if you run against several Kube clusters, make sure you select the right one with:

$ kubectl config get-contexts
...
$ kubectl config set-context
$ kubectl config current-context

The toolkit doesn't yet support specifying a context but this could be requested.

[arun-gupta]

@Lawouach kubectl get pods work fine as shown above. There is only one context explicitly set using KUBECONFIG:

310-chaos-engineering $ kubectl config get-contexts
CURRENT   NAME      CLUSTER      AUTHINFO   NAMESPACE
*         aws       kubernetes   aws        
310-chaos-engineering $ kubectl config current-context
aws

Is system:anonymous user used by kubectl or chaos CLI?

[Lawouach]

Well, the chaostoolkit uses the ~/.kube/config by default, unless you are specific about the credentials. See https://github.com/chaostoolkit/chaostoolkit-kubernetes/blob/master/chaosk8s/__init__.py#L26

But the toolkit does not create or expect any account specifically. It will use whatever it finds. In your case, it seems that it ends up

It's the first time I'm seeing a divergence between kubectl and the official Python client for Kubernetes however. In other words, the toolkit loads the kube config, finds the current context (here "aws") and then uses that. I would be intrigued to see what users kubectl is using, could you run kubectl -v9 get pods (probably don't copy paste the output here for security reasons).

I'm puzzled but it's hard for me to investigate without seeing the actual config file (which obviously you can't share for security reasons ;)).

Side note, you make me realise I should take the context as a parameter to the toolkit Kubernetes extension so it can be specified.

[Lawouach]

May I ask where your kube config is located?

[arun-gupta]

By default, if no KUBECONFIG environment variable is defined then ./kube/config is the right location. But otherwise, the configuration file is read from the location defined by KUBECONFIG. For example, in my case it is:

$ echo $KUBECONFIG 
/Users/argu/tools/eks/kubeconfig

Unset KUBECONFIG, moved kubeconfig to ~/.kube/config, ran the experiment again and got the same error.

More details on KUBECONFIG at https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable

[Lawouach]

Grumble :) I'm puzzled Oo

I really need to understand why we have different behaviors between the kubectl cli and chaostoolkit. First time I notice one.

[arun-gupta]

Let me know what you find out. Happy to have time together to debug it. Would really like to get this working on EKS.

[Lawouach]

So would I :) Thanks for being patient Arun.

[Lawouach]

Hey Arun,

I have pushed a new release of the chaostoolkit-kubernetes extension so that we can specify the context. Now, since you have a single context (as per your log above it seems), this shouldn't make much difference but if you don't mind humouring me :)

$ pip install -U chaostoolkit-kubernetes
$ pip freeze | grep chaostoolkit-kubernetes
0.16.0
$ export KUBERNETES_CONTEXT=aws
$ chaos run ....

Then, if you could send me the chaostoolkit.log file, that'd be great.

[christopherhein]

@Lawouach does the kubernetes client that chaostoolkit-kubernetes uses support the 1.10 ExecCredential flow - https://kubernetes.io/docs/admin/authentication/#client-go-credential-plugins?

If not that could be it, it looks like it's authenticating at system:anonymous on the cluster and EKS is using that to pass a token through to the apiserver.

[arun-gupta]

$ pip freeze | grep chaostoolkit-kubernetes
chaostoolkit-kubernetes==0.16.0

Attaching the log file
chaostoolkit.log

@Lawouach I think the error is more to do with what @christopherhein suggested above though.

[christopherhein]

Yup, forgot to send this… I went through the kubernetes-clients/python project and it doesn't have support for it yet, @arun-gupta if you want to get it working you can go the ServiceAccount w/ desired RBAC rules then configure kubeconfig with the token field set to the SA's token rather than use the ExecCredential flow then try it again to rule the client error out.