Exec format error on arm64

[mycrEEpy]

When using the image registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14 on arm64 nodes the pod crash loops with the following error: exec /node-problem-detector: exec format error

Pod manifest:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-08-30T11:10:36Z"
  generateName: node-problem-detector-
  labels:
    app: node-problem-detector
    controller-revision-hash: 757d568cb6
    pod-template-generation: "2"
  name: node-problem-detector-7z8x7
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: node-problem-detector
    uid: 513debd7-5cc9-435a-b56e-9834abbe0601
  resourceVersion: "21761903"
  uid: 64b29b28-ea07-450f-ad85-befbce85e5ec
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - REDACTED
  containers:
  - command:
    - /node-problem-detector
    - --logtostderr
    - --enable-k8s-exporter
    - --prometheus-address=0.0.0.0
    - --prometheus-port=20257
    - --config.system-log-monitor=/config/kernel-monitor.json
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14
    imagePullPolicy: IfNotPresent
    name: node-problem-detector
    resources:
      limits:
        cpu: 100m
        memory: 80Mi
      requests:
        cpu: 10m
        memory: 80Mi
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/log
      name: log
      readOnly: true
    - mountPath: /dev/kmsg
      name: kmsg
      readOnly: true
    - mountPath: /etc/localtime
      name: localtime
      readOnly: true
    - mountPath: /config
      name: config
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nn5zh
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: REDACTED
  preemptionPolicy: PreemptLowerPriority
  priority: 1010
  priorityClassName: cluster-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: node-problem-detector
  serviceAccountName: node-problem-detector
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  volumes:
  - hostPath:
      path: /var/log/
      type: ""
    name: log
  - hostPath:
      path: /dev/kmsg
      type: ""
    name: kmsg
  - hostPath:
      path: /etc/localtime
      type: ""
    name: localtime
  - configMap:
      defaultMode: 420
      name: node-problem-detector-config-4f59b72b9k
    name: config
  - name: kube-api-access-nn5zh
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-08-30T11:10:36Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-08-30T11:10:36Z"
    message: 'containers with unready status: [node-problem-detector[]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-08-30T11:10:36Z"
    message: 'containers with unready status: [node-problem-detector[]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-08-30T11:10:36Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://1cb1eee4d34c47ec504b311dcafb7e9d760291c6736b292b9191a528de6645de
    image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14
    imageID: registry.k8s.io/node-problem-detector/node-problem-detector@sha256:76da9ce953a0d50e9e14fa29acdc44852a828710c49af6df04ee131cf3c0c7e2
    lastState:
      terminated:
        containerID: containerd://1cb1eee4d34c47ec504b311dcafb7e9d760291c6736b292b9191a528de6645de
        exitCode: 1
        finishedAt: "2023-08-30T11:13:43Z"
        reason: Error
        startedAt: "2023-08-30T11:13:43Z"
    name: node-problem-detector
    ready: false
    restartCount: 5
    started: false
    state:
      waiting:
        message: back-off 2m40s restarting failed container=node-problem-detector
          pod=node-problem-detector-7z8x7_kube-system(64b29b28-ea07-450f-ad85-befbce85e5ec)
        reason: CrashLoopBackOff
  hostIP: REDACTED
  phase: Running
  podIP: REDACTED
  podIPs:
  - ip: REDACTED
  qosClass: Burstable
  startTime: "2023-08-30T11:10:36Z"

[hakman]

I can confirm this:

% docker run -it --rm --entrypoint sh --platform linux/arm64 registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14 

# apt update && apt -y install file       
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main arm64 Packages [8071 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main arm64 Packages [241 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [14.9 kB]                                                                                                                    
...

# file node-problem-detector
node-problem-detector: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=bd9bd59069612b94e84e1c60d73ebf61d618cd44, for GNU/Linux 3.2.0, with debug_info, not stripped

# file bin/ls
bin/ls: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=9c1a4999161b8b2da681b80d8bf351e40afc40ad, for GNU/Linux 3.7.0, stripped

/kind bug

[stafot]

Same experience here

[wangqu123456]

Same experience here

have you solve this problem?

[orondon]

Hi, it's been a while. Has anyone been able to work around this error?
exec /bin/sh: exec format error
I should mention I am trying to use https://catalog.redhat.com/software/containers/openshift4/ose-node-problem-detector-rhel8/5f6d3150bddcd95b309cdffd?container-tabs=dockerfile on Openshift Container Platform 4.14