[connectivity-tests] - Throws segmentation violation -> "invalid memory address or nil pointer dereference"

[tchellomello]

Bug report

General Information

Running the cilium connectivity test traceback with invalid memory address:

$ cilium connectivty test -n cilium-system
[....]
[=] [cilium-test-1] Test [client-egress-to-cidrgroup-deny] [50/105]
......
[=] [cilium-test-1] Test [client-egress-to-cidr-deny-default] [51/105]
......
[=] [cilium-test-1] Skipping test [clustermesh-endpointslice-sync] [52/105] (skipped by condition)
[=] [cilium-test-1] Test [health] [53/105]
....
[=] [cilium-test-1] Skipping test [north-south-loadbalancing] [54/105] (Feature node-without-cilium is disabled)
[=] [cilium-test-1] Test [pod-to-pod-encryption] [55/105]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3b8 pc=0x3039042]

goroutine 39573 [running]:
github.com/cilium/cilium/cilium-cli/connectivity/check.Pod.Address({0x0, 0x0, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0}, 0x1)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/peer.go:107 +0x62
github.com/cilium/cilium/cilium-cli/connectivity/tests.getInterNodeIface({0x55f1af0, 0xc0000df940}, 0xc0007f6c60, 0xc002a0a2c0, 0xc002a0a340, 0xc002a0a280, 0xc002a0a300, 0x1, 0x0)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:54 +0x308
github.com/cilium/cilium/cilium-cli/connectivity/tests.testNoTrafficLeak({0x55f1af0, 0xc0000df940}, 0xc0007f6c60, {0x55cd218, 0xc000fbb6e0}, 0xc002a0a280, 0xc002a0a2c0, 0xc002a0a300, 0xc002a0a340, 0x0, ...)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:303 +0x2bb
github.com/cilium/cilium/cilium-cli/connectivity/tests.(*podToPodEncryption).Run.func1(0x1)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:279 +0x1a6
github.com/cilium/cilium/cilium-cli/connectivity/check.(*Test).ForEachIPFamily(0xc0007f6c60, 0xc002dcdc40)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/test.go:840 +0xd8
github.com/cilium/cilium/cilium-cli/connectivity/tests.(*podToPodEncryption).Run(0xc000fbb6e0, {0x55f1af0, 0xc0000df940}, 0xc0007f6c60)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:274 +0x5fa
github.com/cilium/cilium/cilium-cli/connectivity/check.(*Test).Run(0xc0007f6c60, {0x55f1af0, 0xc0000df940}, 0x37)
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/test.go:375 +0x664
github.com/cilium/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run.func1()
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/context.go:411 +0x99
created by github.com/cilium/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run in goroutine 894
	/go/src/github.com/cilium/cilium-cli/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/context.go:405 +0x91

 ....

Implementation details:

• Wireguard encryption enabled
• Wireguard node-to-node encryption enabled
• KubeProxy replacement

• Cilium CLI version (run cilium version)

$ cilium version -n cilium-system 
cilium-cli: v0.16.20 compiled with go1.23.2 on linux/amd64
cilium image (default): v1.16.3
cilium image (stable): v1.16.3
cilium image (running): v1.16.3@sha256:62d2a09bbef840a46099ac4c69421c90f84f28d018d479749049011329aa7f28

• Orchestration system version in use (e.g. kubectl version, ...)

$ kubectl version
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.28.13-eks-a737599

• Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions)

EKS

• Generate and upload a system zip: cilium sysdump

How to reproduce the issue

1. Deploy an EKS cluster without any CNI;
2. Deploy Cilium with the following options:

bpf:
  hostLegacyRouting: false
  masquerade: true

clustermesh:
  replicas: 1
  useAPIServer: false
  apiserver:
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      type: LoadBalancer
    tls:
      enabled: true
      auto:
        enabled: true
        method: cronJob

encryption:
  enabled: true
  nodeEncryption: true
  type: wireguard

eni:
  awsEnablePrefixDelegation: true
  enabled: true

envoy:
  enabled: true
  rollOutPods: true

externalWorkloads:
  enabled: false

gatewayAPI:
  enabled: false

ipam:
  mode: eni

hubble:
  enabled: false
  relay:
    enabled: false
  tls:
    enabled: true
    auto:
      enabled: true
      method: cronJob
operator:
  prometheus:
    enabled: false
  replicas: 1
  unmanagedPodWatcher:
    restart: true

devices: eth0
routingMode: native
k8sServiceHost: <KUBE_API_HOST>
k8sServicePort: 443
kubeProxyReplacement: true

3. Run cilium connectivty test

[tchellomello]

Same problem on different test

[=] [cilium-test-1] Skipping test [clustermesh-endpointslice-sync] [52/105] (skipped by condition)
[=] [cilium-test-1] Test [health] [53/105]
.....panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3b8 pc=0x3039042]

goroutine 47467 [running]:
github.com/cilium/cilium/cilium-cli/connectivity/check.Pod.Address({0x0, 0x0, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0}, 0x1)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/peer.go:107 +0x62
github.com/cilium/cilium/cilium-cli/connectivity/tests.getInterNodeIface({0x55f1af0, 0xc000538d80}, 0xc0012f4c60, 0xc0018cb580, 0xc0018cb600, 0xc0018cb5c0, 0xc0018cb640, 0x1, 0x0)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:54 +0x308
github.com/cilium/cilium/cilium-cli/connectivity/tests.testNoTrafficLeak({0x55f1af0, 0xc000538d80}, 0xc0012f4c60, {0x55cd218, 0xc0012379f8}, 0xc0018cb580, 0xc0018cb5c0, 0xc0018cb600, 0xc0018cb640, 0x0, ...)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:288 +0xec
github.com/cilium/cilium/cilium-cli/connectivity/tests.(*podToPodEncryption).Run.func1(0x1)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:279 +0x1a6
github.com/cilium/cilium/cilium-cli/connectivity/check.(*Test).ForEachIPFamily(0xc0012f4c60, 0xc000821c40)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/test.go:840 +0xd8
github.com/cilium/cilium/cilium-cli/connectivity/tests.(*podToPodEncryption).Run(0xc0012379f8, {0x55f1af0, 0xc000538d80}, 0xc0012f4c60)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/tests/encryption.go:274 +0x5fa
github.com/cilium/cilium/cilium-cli/connectivity/check.(*Test).Run(0xc0012f4c60, {0x55f1af0, 0xc000538d80}, 0x37)
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/test.go:375 +0x664
github.com/cilium/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run.func1()
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/context.go:411 +0x99
created by github.com/cilium/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run in goroutine 1219
	/cilium/vendor/github.com/cilium/cilium/cilium-cli/connectivity/check/context.go:405 +0x91

[tchellomello]

Added a few comments on the code and it seems the serverHost *checkPod is already coming with a wrong address

$ ./cilium  connectivity test -n cilium-system --verbose --debug --test pod-to-pod-encryption                                                                                              
  🐛 Finalizing Test north-south-loadbalancing
  [-] Scenario [pod-to-pod-encryption/pod-to-pod-encryption]
 
** added a spew on `peer.go`  we can see the object

(string)ip-10-25-46-94.ec2.internal
(check.Pod)(PANIC=runtime error: invalid memory address or nil pointer dereference){K8sClient:(*k8s.Client)<nil> Pod:(*v1.Pod)<nil> scheme:(string) path:(string) port:(uint32)0 Outside:(bool)false}(check.Pod)(PANIC=runtime error: invalid memory address or nil pointer dereference){K8sClient:(*k8s.Client)<nil> Pod:(*v1.Pod)<nil> scheme:(string) path:(string) port:(uint32)0 Outside:(bool)false}([]v1.PodIP)[&PodIP{IP:10.25.42.206,}]([]v1.PodIP)[&PodIP{IP:10.25.20.133,}]([]v1.PodIP)[&PodIP{IP:10.25.42.206,}]([]v1.PodIP)[&PodIP{IP:10.25.20.133,}]  


  🐛 mmello -> srcIP: 10.25.42.206
  🐛 mmello -> dstIP: 10.25.20.133
  🐛 mmello -> ipRouteGetCmd(1): ip -o route get 10.25.20.133 from 10.25.42.206 iif cilium_host

** then here accessing the `serverHost` raises the trace

%!s(PANIC=String method: runtime error: invalid memory address or nil pointer dereference) 

  🐛 Finalizing Test pod-to-pod-encryption
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x3b8 pc=0x10742be68]

  42 func getInterNodeIface(ctx context.Context, t *check.Test,
  43     client, clientHost, server, serverHost *check.Pod, ipFam features.IPFamily,
  44     wgEncap bool) string {
  45
  46     tunnelEnabled := false
  47     tunnelMode := ""
  48     if tunnelFeat, ok := t.Context().Feature(features.Tunnel); ok && tunnelFeat.Enabled {
  49         tunnelEnabled = true
  50         tunnelMode = tunnelFeat.Mode
  51     }
  52
  53     srcIP, dstIP := client.Address(ipFam), server.Address(ipFam)
  54     t.Debugf("mmello -> srcIP: %s", srcIP)
  55     t.Debugf("mmello -> dstIP: %s", dstIP)
  56
  57     //ipRouteGetCmd := fmt.Sprintf("ip -o route get %s from %s", dstIP, srcIP)
  58     ipRouteGetCmd := fmt.Sprintf("ip -o route get %s from %s iif cilium_host", dstIP,          srcIP)
  59     t.Debugf("mmello -> ipRouteGetCmd(1): %s", ipRouteGetCmd)
  60
  61     spew.Printf("%s", serverHost)