Intel binary present on the ARM

[Marta-Panfilova-ASL]

We are getting following error from NPD in our nodes /bin/sh: 1: exec: /node-problem-detector: Exec format error

Crashing because of Intel/AMD 64bit binary present on aarch64/ARM64 docker image.

root@:/# file node-problem-detector node-problem-detector: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=bd9bd59069612b94e84e1c60d73ebf61d618cd44, for GNU/Linux 3.2.0, with debug_info, not stripped

root@:/# unae me -a Linux 654213c1a942 6.4.16-orbstack-00105-g14094bfeec09 #1 SMP Mon Sep 18 21:45:38 UTC 2023 aarch64 GNU/Linux

root@:/# file /bin/bash /bin/bash: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=96c22dfc3c74ccf4ba77d9cce6fc2c5e635456c1, for GNU/Linux 3.7.0, stripped

[autarchprinceps]

I get the same issue with at the least registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14
Seems like you guys didn't build the binary on each architecture this time. This used to work, but I couldn't quite say which version it broke at. Maybe it was more a change in your CI system too, unrelated to the particular version. Definitively a straight up bug, this image will not work like this on ARM. It still works for x86 of course, but like this there isn't a point to building it multiarch for ARM at all, which it is:

Name:   registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:76da9ce953a0d50e9e14fa29acdc44852a828710c49af6df04ee131cf3c0c7e2
 * Contains 2 manifest references (2 images, 0 attestation):
[1]     Type: application/vnd.docker.distribution.manifest.v2+json
[1]   Digest: sha256:98034f47cdfa331105265f025c0dbff781e8c15436f726eceacec52644a7c014
[1]   Length: 1488
[1] Platform:
[1]    -      OS: linux
[1]    -    Arch: amd64
[1] # Layers: 6
     layer 01: digest = sha256:c4e25f9975ade78ed0838b6cbf3f4fe1c6bf817811f3c8e676b1cbb6a7104950
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 02: digest = sha256:22cdf6360626d401a884ae910323267a7d7876e5717ed11d9d13bbc6cf8cb0db
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 03: digest = sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 04: digest = sha256:a19a5af3e67a96f03e92287598725e391cf7df49799ceb0f1abc1c88fdfb8e82
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 05: digest = sha256:f239dff246d22a7de3be61ff3679722e66982089630b0591a057d01889f0c986
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 06: digest = sha256:cad51d6b0d0b6b9ac3b3f1a0051eb93e35d1b84c5e25ad2482f8b71a49e58a01
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip

[2]     Type: application/vnd.docker.distribution.manifest.v2+json
[2]   Digest: sha256:38ddabcad5a96bfb4af3bf6e61348198f7da4441b08a17c9ad0ea9bdb3a8689a
[2]   Length: 1488
[2] Platform:
[2]    -      OS: linux
[2]    -    Arch: arm64
[2] # Layers: 6
     layer 01: digest = sha256:f91cc52e2c97c480189bb2da87653dd9abfe2679c36fc41b9db68878827748b4
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 02: digest = sha256:86c650b37460bdd568db653fe0ea8d628fe802bb15b47c5c2879fb2fd80b1d50
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 03: digest = sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 04: digest = sha256:a19a5af3e67a96f03e92287598725e391cf7df49799ceb0f1abc1c88fdfb8e82
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 05: digest = sha256:f239dff246d22a7de3be61ff3679722e66982089630b0591a057d01889f0c986
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip
     layer 06: digest = sha256:cad51d6b0d0b6b9ac3b3f1a0051eb93e35d1b84c5e25ad2482f8b71a49e58a01
                 type = application/vnd.docker.image.rootfs.diff.tar.gzip

[runningman84]

The problem does not exist for older versions like registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.13

[hakman]

This was fixed in #801, just that a new release will be needed.

[runningman84]

@hakman do you have any ETA for the release? We use flux to auto apply minor updates and this broke a lot of clusters...

[hakman]

There are very few people that can do releases. This is why #819.

[disconn3ct]

Are you saying that releases on hold until that feature ticket is resolved? This fix has been accepted, there shouldn't be much of a process block. Releases happened before somehow, and this is a major breakage.

[tuxpeople]

I would suggest thinking about fixing the release process, as a release process not allowing to publish an important fix for a major breakage for two weeks is undoubtedly broken.

[disconn3ct]

Two months without a release. This takes "breaking the deploy and going home" to a whole new level.

[hakman]

@vteratipally @Random-Liu Can we do something about a NPD release?

[hakman]

NPD v0.8.15 has been released:

% crane digest registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.15
sha256:15a8bad79ae26124109e6fdb696fd8536b2043f4ece8ff48f03073157d6a2c2d

/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

NPD v0.8.15 has been released:

% crane digest registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.15
sha256:15a8bad79ae26124109e6fdb696fd8536b2043f4ece8ff48f03073157d6a2c2d

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vteratipally]

@hakman

Could you please publish the release notes

[hakman]

Could you please publish the release notes

@vteratipally I think only approvers have permission to publish releases.

[disconn3ct]

NPD v0.8.15 has been released:

Is it a release if it is only partially done? Can this be opened until the release is actually complete?