busyboy with version greater than 1.28 seem to have a bug. Several users of Kubernetes reported, that the issues still exists. See: https://github.com/docker-library/busybox/issues/48 The issue was reported several months ago, but the fix did not solve the issue: https://bugs.busybox.net/show_bug.cgi?id=11161
you can reproduce it like this: # install kind: https://kind.sigs.k8s.io/docs/user/quick-start/ kind create cluster guettli@p15:~/.kube$ cat busy-new.yaml apiVersion: v1 kind: Pod metadata: name: busybox-new namespace: default spec: containers: - name: busybox image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always guettli@p15:~/.kube$ kubectl apply -f busy-new.yaml guettli@p15:~/.kube$ kubectl exec -ti busybox-new -- nslookup kubernetes.default Server: 10.96.0.10 Address: 10.96.0.10:53 ** server can't find kubernetes.default: NXDOMAIN ---------- If you use "busybox:1.28" as image, it works fine.
A simpler reproducer (that does not require Kubernetes) is something like this: $ docker run --rm --pull=always --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=google.com --dns 1.1.1.1 busybox:uclibc nslookup www ... ** server can't find www.f1: NXDOMAIN ... If you drop the number of search domains down to just three, it works: $ docker run --rm --pull=always --dns-search=f1 --dns-search=f2 --dns-search=google.com --dns 1.1.1.1 busybox:uclibc nslookup www ... Non-authoritative answer: Name: www.google.com Address: 142.250.176.4 ... This was on BusyBox version 1.34.1 built against uClibc-ng via buildroot. Interestingly, it does *not* reproduce if I switch to "busybox:glibc" (built against Debian's glibc) or "busybox:musl" (built against Alpine's musl).
Can you confirm whether this is still related to concurrent issue of A and AAAA lookup as per https://bugs.busybox.net/show_bug.cgi?id=11161#c4 ? For the prior bug, specifying -type=a consistently removed the intermittent failure (which presented when there was no AAAA record).
Just for the recors, in the past there have been DNS issues with other libc implementations. Maybe this helps to solve the uclibc-ng issue: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues