Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Commit e0cef57

Browse files
garylouggaryloug
committed
Add l3fwd steps to README.md
Include steps in the OVS-DPDK section of the README that tell the user how to spin up a l3fwd pod and a pktgen pod. Pktgen sends traffic to the l3fwd pod where it is forwarded back to pktgen. Pktgen can display some statistics on the traffic. Signed-off-by: Gary Loughnane <[email protected]>
1 parent 374f99c commit e0cef57

File tree

1 file changed

+233
-0
lines changed

1 file changed

+233
-0
lines changed

Diff for: ‎README.md

+233
Original file line numberDiff line numberDiff line change
@@ -762,6 +762,239 @@ by getting the port numbers with `ovs-ofctl dump-ports br0` and configuring
762762
flow, for example, from port 1 to port 2 with
763763
`ovs-ofctl add-flow br0 in_port=1,action=output:2` and vice versa.
764764

765+
## Testing with DPDK L3FWD Application
766+
767+
To follow this example you should have a system with kubernetes available and configured to support native 1 GB hugepages. You should also have multus-cni and userspace-cni-network-plugin up and running. See `examples/crd-userspace-net-ovs-no-ipam.yaml` for example config to use with multus. If using OVS, check that you have bridge named `br0` in your OVS with `ovs-vsctl show` and if not, create it with `ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev`.
768+
769+
### 1. Build the images to be used
770+
771+
Create the l3fwd Dockerfile:
772+
```bash
773+
$ cat <<EOF > l3fwd-dockerfile
774+
FROM ubuntu:bionic
775+
776+
RUN apt-get update && apt-get install -y --no-install-recommends \\
777+
build-essential make wget vim dpdk libnuma-dev \\
778+
&& apt-get clean && rm -rf /var/lib/apt/lists/*
779+
780+
ENV DPDK_VERSION=17.11.3 \\
781+
RTE_SDK=/usr/src/dpdk \\
782+
RTE_TARGET=x86_64-native-linuxapp-gcc
783+
784+
RUN wget http://fast.dpdk.org/rel/dpdk-\${DPDK_VERSION}.tar.xz && tar xf dpdk-\${DPDK_VERSION}.tar.xz && \\
785+
mv dpdk-stable-\${DPDK_VERSION} \${RTE_SDK}
786+
787+
RUN sed -i s/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/ \${RTE_SDK}/config/common_linuxapp \\
788+
&& sed -i s/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/ \${RTE_SDK}/config/common_linuxapp \\
789+
&& sed -i s/CONFIG_RTE_KNI_KMOD=y/CONFIG_RTE_KNI_KMOD=n/ \${RTE_SDK}/config/common_linuxapp
790+
791+
RUN cd \${RTE_SDK} && make install T=\${RTE_TARGET} && make -C examples
792+
793+
WORKDIR \${RTE_SDK}/examples/l3fwd/\${RTE_TARGET}/app/
794+
EOF
795+
```
796+
797+
Build the l3fwd Docker image, tag it as `dpdk-l3fwd`:
798+
```bash
799+
docker build -t dpdk-l3fwd -f l3fwd-dockerfile .
800+
```
801+
802+
Create the pktgen Dockerfile:
803+
```bash
804+
$ cat <<EOF > pktgen-dockerfile
805+
FROM debian:jessie
806+
807+
RUN apt-get update && apt-get install -y --no-install-recommends \\
808+
gcc build-essential make wget curl git liblua5.2-dev libedit-dev libpcap-dev libncurses5-dev libncursesw5-dev pkg-config vim libnuma-dev ca-certificates \\
809+
&& apt-get clean && rm -rf /var/lib/apt/lists/*
810+
811+
ENV DPDK_VERSION=17.11.3 \\
812+
RTE_SDK=/usr/src/dpdk \\
813+
RTE_TARGET=x86_64-native-linuxapp-gcc \\
814+
PKTGEN_COMMIT=pktgen-3.4.8 \\
815+
PKTGEN_DIR=/usr/src/pktgen
816+
817+
RUN wget http://fast.dpdk.org/rel/dpdk-\${DPDK_VERSION}.tar.xz && tar xf dpdk-\${DPDK_VERSION}.tar.xz && \\
818+
mv dpdk-stable-\${DPDK_VERSION} \${RTE_SDK}
819+
820+
RUN sed -i s/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/ \${RTE_SDK}/config/common_linuxapp \\
821+
&& sed -i s/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/ \${RTE_SDK}/config/common_linuxapp \\
822+
&& sed -i s/CONFIG_RTE_KNI_KMOD=y/CONFIG_RTE_KNI_KMOD=n/ \${RTE_SDK}/config/common_linuxapp
823+
824+
RUN sed -i s/CONFIG_RTE_APP_TEST=y/CONFIG_RTE_APP_TEST=n/ \${RTE_SDK}/config/common_linuxapp \\
825+
&& sed -i s/CONFIG_RTE_TEST_PMD=y/CONFIG_RTE_TEST_PMD=n/ \${RTE_SDK}/config/common_linuxapp
826+
827+
RUN cd \${RTE_SDK} \\
828+
&& make install T=\${RTE_TARGET} DESTDIR=install -j
829+
830+
RUN git config --global user.email "root@container" \\
831+
&& git config --global user.name "root"
832+
833+
RUN git clone http://dpdk.org/git/apps/pktgen-dpdk \\
834+
&& cd pktgen-dpdk \\
835+
&& git checkout \${PKTGEN_COMMIT} \\
836+
&& cd .. \\
837+
&& mv pktgen-dpdk \${PKTGEN_DIR}
838+
839+
RUN cd \${PKTGEN_DIR} \\
840+
&& make -j
841+
842+
WORKDIR \${PKTGEN_DIR}/
843+
EOF
844+
```
845+
846+
Build the pktgen Docker image, tag it as`dpdk-pktgen`:
847+
```bash
848+
docker build -t dpdk-pktgen -f pktgen-dockerfile .
849+
```
850+
851+
### 2. Create the pods to be used
852+
853+
Create the l3fwd Pod Spec:
854+
```bash
855+
$ cat <<EOF > l3fwd-pod.yaml
856+
apiVersion: v1
857+
kind: Pod
858+
metadata:
859+
generateName: dpdk-l3fwd-
860+
annotations:
861+
k8s.v1.cni.cncf.io/networks: userspace-networkobj
862+
spec:
863+
containers:
864+
- name: dpdk-l3fwd
865+
image: dpdk-l3fwd
866+
imagePullPolicy: IfNotPresent
867+
securityContext:
868+
privileged: true
869+
runAsUser: 0
870+
volumeMounts:
871+
- mountPath: /vhu/
872+
name: socket
873+
- mountPath: /dev/hugepages
874+
name: hugepage
875+
resources:
876+
requests:
877+
memory: 2Gi
878+
limits:
879+
hugepages-1Gi: 2Gi
880+
command: ["sleep", "infinity"]
881+
volumes:
882+
- name: socket
883+
hostPath:
884+
path: /var/lib/cni/vhostuser/
885+
- name: hugepage
886+
emptyDir:
887+
medium: HugePages
888+
securityContext:
889+
runAsUser: 0
890+
restartPolicy: Never
891+
EOF
892+
```
893+
894+
Create the pktgen Pod Spec:
895+
```bash
896+
$ cat <<EOF > pktgen-pod.yaml
897+
apiVersion: v1
898+
kind: Pod
899+
metadata:
900+
generateName: dpdk-pktgen-
901+
annotations:
902+
k8s.v1.cni.cncf.io/networks: userspace-networkobj
903+
spec:
904+
containers:
905+
- name: dpdk-pktgen
906+
image: dpdk-pktgen
907+
imagePullPolicy: IfNotPresent
908+
securityContext:
909+
privileged: true
910+
runAsUser: 0
911+
volumeMounts:
912+
- mountPath: /vhu/
913+
name: socket
914+
- mountPath: /dev/hugepages
915+
name: hugepage
916+
resources:
917+
requests:
918+
memory: 2Gi
919+
limits:
920+
hugepages-1Gi: 2Gi
921+
command: ["sleep", "infinity"]
922+
volumes:
923+
- name: socket
924+
hostPath:
925+
path: /var/lib/cni/vhostuser/
926+
- name: hugepage
927+
emptyDir:
928+
medium: HugePages
929+
securityContext:
930+
runAsUser: 0
931+
restartPolicy: Never
932+
EOF
933+
```
934+
935+
Deploy both pods:
936+
```bash
937+
$ kubectl create -f l3fwd-pod.yaml
938+
$ kubectl create -f pktgen-pod.yaml
939+
```
940+
941+
Verify your pods are running and note their unique name IDs:
942+
```bash
943+
$ kubectl get pods
944+
NAME READY STATUS RESTARTS AGE
945+
dpdk-l3fwd-vj4hj 1/1 Running 0 51s
946+
dpdk-pktgen-xrsz9 1/1 Running 0 2s
947+
```
948+
949+
### 3. Run L3FWD and Pktgen
950+
951+
Using your pod ID, open a bash shell in the pktgen pod:
952+
```bash
953+
$ kubectl exec -it dpdk-pktgen-<ID> bash
954+
```
955+
956+
Export the port ID prefex and start the pktgen application (adjust cores, memory, etc. to suit your system):
957+
```bash
958+
$ export ID=$(/vhu/get-prefix.sh)
959+
$ ./app/x86_64-native-linuxapp-gcc/pktgen -l 10,11,12 --vdev=virtio_user0,path=/vhu/${ID}/${ID:0:12}-net1 --no-pci --socket-mem=1024,1024 --master-lcore 10 -- -m 11:12.0 -P
960+
```
961+
962+
The pktgen application should launch. Among the statistics, make note of the pktgen source MAC address listed as ```Src MAC Address```. For example:
963+
```bash
964+
Src MAC Address : f2:89:22:4e:28:3b
965+
```
966+
967+
In another terminal, open a bash shell in the l3fwd pod:
968+
```bash
969+
kubectl exec -it dpdk-l3fwd-<ID> bash
970+
```
971+
972+
Export the port ID prefex and start the l3fwd application. Set the destination MAC address using the ```--eth-dest``` argument. This should be the ```Src MAC Address``` previously noted from the pktgen pod (adjust cores, memory, etc. to suit your system):
973+
```bash
974+
$ export ID=$(/vhu/get-prefix.sh)
975+
$ ./l3fwd -c 0x10 --vdev=virtio_user0,path=/vhu/${ID}/${ID:0:12}-net1 --no-pci --socket-mem=1024,1024 -- -p 0x1 -P --config "(0,0,4)" --eth-dest=0,<pktgen-source-mac-add> --parse-ptype
976+
```
977+
978+
The l3fwd app should start up. Among the information printed to the screen will be the ```Address```. This is the MAC address of the l3fwd port, make note of it.
979+
980+
Back on the pktgen pod, set the destination MAC address to that of the l3fwd port:
981+
```bash
982+
Pktgen:/> set 0 dst mac <l3fwd-mac-address>
983+
```
984+
985+
Start traffic generation:
986+
```bash
987+
Pktgen:/> start 0
988+
```
989+
990+
You should see the packet counts for Tx and Rx increase, verifying that packets are being transmitted by pktgen and are being sent back via l3fwd running in the other pod.
991+
992+
To exit:
993+
```bash
994+
Pktgen:/> stop 0
995+
Pktgen:/> quit
996+
```
997+
765998

766999
# Contacts
7671000
For any questions about Userspace CNI, please reach out on github issue or feel free to contact the developer @Kural, @abdul or @bmcfall in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/)

0 commit comments

Comments
 (0)