Azure NPM troubleshooting guide

Steps for updating Azure-NPM manually:

1. Save all network policies applied to your cluster

mkdir -p networkpolicies
kubectl get networkpolicy --no-headers=true --all-namespaces | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 get networkpolicy \2 -o yaml --export > networkpolicies\/\2.yaml/e'

2. Delete all network policies applied to your cluster (Make sure you have saved them first!)

kubectl get networkpolicy --no-headers=true --all-namespaces | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete networkpolicy \2/e'

3. Backup iptables and ipset

sudo iptables-save > iptables_backup
sudo ipset save > ipset_backup

4. Flush iptables entries generated by Azure-NPM (regardless of version)

sudo iptables -F AZURE-NPM
sudo iptables -X AZURE-NPM
sudo iptables -F AZURE-NPM-INGRESS-FROM-NS
sudo iptables -X AZURE-NPM-INGRESS-FROM-NS
sudo iptables -F AZURE-NPM-INGRESS-FROM-POD
sudo iptables -X AZURE-NPM-INGRESS-FROM-POD
sudo iptables -F AZURE-NPM-EGRESS-TO-NS
sudo iptables -X AZURE-NPM-EGRESS-TO-NS
sudo iptables -F AZURE-NPM-EGRESS-TO-POD
sudo iptables -X AZURE-NPM-EGRESS-TO-POD
sudo iptables -F AZURE-NPM-INGRESS-FROM
sudo iptables -X AZURE-NPM-INGRESS-FROM
sudo iptables -F AZURE-NPM-EGRESS-TO
sudo iptables -X AZURE-NPM-EGRESS-TO
sudo iptables -F AZURE-NPM-INGRESS-PORT
sudo iptables -X AZURE-NPM-INGRESS-PORT
sudo iptables -F AZURE-NPM-EGRESS-PORT
sudo iptables -X AZURE-NPM-EGRESS-PORT
sudo iptables -F AZURE-NPM-TARGET-SETS
sudo iptables -X AZURE-NPM-TARGET-SETS
sudo iptables -F AZURE-NPM-KUBE-SYSTEM
sudo iptables -X AZURE-NPM-KUBE-SYSTEM

5. Update Azure-NPM daemonset

kubectl delete ds azure-npm -n kube-system --grace-period=0 --force && \
kubectl apply -f https://raw.githubusercontent.com/Azure/azure-container-networking/master/npm/azure-npm.yaml

6. Re-apply saved network policies from networkpolicies/ directory