when the service port changes, the ingress cannot detect the change and adjust the upstream port. #12209

zzzzoy · 2024-10-18T10:16:46Z

What happened:
At first, I used the wrong targetPort in the service, created the associated ingress but could not be accessed, so I fixed the targetPort of the service. I think the ingress should reload the service to ensure the correct forwarding port. But the fact is that the ingress won't reload and still accesses the wrong targetPort. I had to rebuild the ingress.yaml to make sure it work properly.
I wonder, is this a normal situation? or ingress cannot supported this feature for some reason

To illustrate this problem in a simple example,

I created an nginx service, and used the wrong targetPort port：

service.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-demo
  name: nginx-demo
  namespace: test
spec:
  internalTrafficPolicy: Cluster
  ports:
  - name: service-0
    port: 80
    protocol: TCP
    targetPort: 81                      <-------------------- wrong port
  selector:
    app: nginx-demo
  type: ClusterIP

create ingress for this service, now we can try to access www.test.com but ingress will response 502 Bad Gateway

ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test
  namespace: test
spec:
  rules:
  - host: www.test.com
    http:
      paths:
      - backend:
          service:
            name: nginx-demo
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

modify the targetPort of service and apply kubectl apply -f service.yaml

service.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-demo
  name: nginx-demo
  namespace: test
spec:
  internalTrafficPolicy: Cluster
  ports:
  - name: service-0
    port: 80
    protocol: TCP
    targetPort: 80                      <-------------------- fixed port
  selector:
    app: nginx-demo
  type: ClusterIP

try curl ingress still response 502 Bad Gateway. But we curl service will get response properly

curl -k www.test.com

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

clusterIP=`kubectl get svc nginx-demo -ntest -o=jsonpath='{.spec.clusterIP}'`

curl -k $clusterIP
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

to rebuild ingress by kubectl delete -f ingress.yaml && kubectl apply -f ingress.yaml and so far we can get correct response message from ingress

What you expected to happen:
do not need to rebuild ingress, ingress will reload service change by itself

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

v1.10.1

Kubernetes version (use kubectl version):
v1.23.15

ingress version:
v1.10.1
OS (e.g. from /etc/os-release): Centos 7.6
Kernel (e.g. uname -a):
Linux 192-168-0-174 3.10.0-862.14.4.el7.x86_64 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
kubeadm
Basic cluster related info:
- kubectl version
- v1.23.15
- kubectl get nodes -o wide
- NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
  192.168.0.121 Ready 3d21h v1.23.15 192.168.0.121 CentOS Linux 7 (Core) 3.10.0-862.14.4.el7.x86_64 docker://18.9.0
  192.168.0.174 Ready etcd,master 4d1h v1.23.15 192.168.0.174 CentOS Linux 7 (Core) 3.10.0-862.14.4.el7.x86_64 docker://18.9.0
  192.168.0.185 Ready etcd,master 4d1h v1.23.15 192.168.0.185 CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.0
  192.168.0.57 Ready etcd,master 4d1h v1.23.15 192.168.0.57 CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.0
How was the ingress-nginx-controller installed: helm
Current State of the controller: running

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2024-10-18T10:16:54Z

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

longwuyuan · 2024-10-18T18:46:16Z

The information you have provided is very very vague as it can not be used for reproducing a problem in a way that helps
You can help the readers by first looking at the template of a new bug report and answer the questions that are asked in the template, here, by editing the issue description of this issue
You can also help by ensuring markdown format because right now its harder to read the data
Looks like you changes some specs and want to see response change but its not even clear if you get a response from a cache like a browser cache
After you have provided the info as suggested above, please try to help the readers here by providing precise instructions that someone can copy/paste from , to reproduce the problem you are having. Do not make it vague like refreshing a browser

Once the information in the issue description is helpful to readers here, then you can re-open this issue. I will close it for now as there is no action item here.

/remove-kind bug
/close

k8s-ci-robot · 2024-10-18T18:46:21Z

@longwuyuan: Closing this issue.

In response to this:

The information you have provided is very very vague as it can not be used for reproducing a problem in a way that helps

You can help the readers by first looking at the template of a new bug report and answer the questions that are asked in the template, here, by editing the issue description of this issue

You can also help by ensuring markdown format because right now its harder to read the data

Looks like you changes some specs and want to see response change but its not even clear if you get a response from a cache like a browser cache

After you have provided the info as suggested above, please try to help the readers here by providing precise instructions that someone can copy/paste from , to reproduce the problem you are having. Do not make it vague like refreshing a browser

Once the information in the issue description is helpful to readers here, then you can re-open this issue. I will close it for now as there is no action item here.

/remove-kind bug
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zzzzoy · 2024-10-28T07:56:06Z

/reopen
@longwuyuan

Hello, thanks for your advice, I made changes to the description and format

k8s-ci-robot · 2024-10-28T07:56:12Z

@zzzzoy: Reopened this issue.

In response to this:

/reopen
@longwuyuan

Hello, thanks for your advice, I made changes to the description and format

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

longwuyuan · 2024-10-28T08:09:42Z

I don't see a description anywhere anywhere that you will change the specs of a backend service to cause a re-synchronization of the ingress resource.

I think you should first check the nginx.conf and confirm that the backend is configured in the related server block for the related location.

Then you should check the logs of the controller to see if the request you sent matched a rule and it it was routed.

If you see both and see a confirmation from both these data points, then that ends the work of the ingress-controller. The ingress-controller uses endpoints to keep track of backends, to which the traffic must be routed. If the backend fails to route because TargetPort changed, I don't think its a problem of the ingress-controller.

In any case what you should test is to send request to a local port that is port-forward to the service. Then change the TargetPort. And send request again.

Wait for comments from other experts, if I am wrong on this.

zzzzoy · 2024-10-28T09:32:54Z

I don't see a description anywhere anywhere that you will change the specs of a backend service to cause a re-synchronization of the ingress resource.

I think you should first check the nginx.conf and confirm that the backend is configured in the related server block for the related location.

Then you should check the logs of the controller to see if the request you sent matched a rule and it it was routed.

If you see both and see a confirmation from both these data points, then that ends the work of the ingress-controller. The ingress-controller uses endpoints to keep track of backends, to which the traffic must be routed. If the backend fails to route because TargetPort changed, I don't think its a problem of the ingress-controller.

In any case what you should test is to send request to a local port that is port-forward to the service. Then change the TargetPort. And send request again.

Wait for comments from other experts, if I am wrong on this.

hello. after i fix the service targetPort from 81 to 80,
i have check the nginx-ingress-controller.log, still use port 81 as upstreaming

2024/10/28 17:26:04 [error] 1162#1162: *3136552 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.166, server: www.test.com, request: "GET / HTTP/1.1", upstream: "http://172.16.243.146:81/", host: "www.test.com"

this is nginx.conf

        ## start server www.test.com                                                                                                           
        server {                                                                                                                               
                server_name www.test.com ;                                                                                                     
                                                                                                                                               
                http2 on;                                                                                                                      
                                                                                                                                               
                listen 80  ;                                                                                                                   
                listen [::]:80  ;                                                                                                              
                listen 443  ssl;                                                                                                               
                listen [::]:443  ssl;                                                                                                          
                                                                                                                                               
                set $proxy_upstream_name "-";                                                                                                  
                                                                                                                                               
                ssl_certificate_by_lua_block {                                                                                                 
                        certificate.call()                                                                                                     
                }                                                                                                                              
                                                                                                                                               
                location / {                                                                                                                   
                                                                                                                                               
                        set $namespace      "test";                                                                                            
                        set $ingress_name   "test";                                                                                            
                        set $service_name   "nginx-demo";                                                                                      
                        set $service_port   "80";                                                                                              
                        set $location_path  "/";                                                                                               
                        set $global_rate_limit_exceeding n;                                                                                    
                                                                                                                                               
                        rewrite_by_lua_block {                                                                                                 
                                lua_ingress.rewrite({                                                                                          
                                        force_ssl_redirect = false,                                                                            
                                        ssl_redirect = true,                                                                                   
                                        force_no_ssl_redirect = false,                                                                         
                                        preserve_trailing_slash = false,                
                                        use_port_in_redirects = false,                                                                         
                                        global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },      
                                })                                                                                                             
                                balancer.rewrite()                                                                                             
                                plugins.run()                                                                                                  
                        }                                                                                                                      
                                                                                                                                               
                        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any                                    
                        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
                        # other authentication method such as basic auth or external auth useless - all requests will be allowed.              
                        #access_by_lua_block {                                                                                                 
                        #}                                                                                                                     
                                                                                                                                               
                        header_filter_by_lua_block {                                                                                           
                                lua_ingress.header()                                                                                           
                                plugins.run()                                                                                                  
                        }                                                                                                                      
                                                                                                                                               
                        body_filter_by_lua_block {                                                                                             
                                plugins.run()                                                                                                  
                        }                                                                                                                      
                                                                                                                                               
                        log_by_lua_block {                                                                                                     
                                balancer.log()                                                                                                 
                                                                                                                                               
                                monitor.call()                                                                                                 
                                                                                                                                               
                                plugins.run()                                                                                                  
                        }                                                                                                                      
                                                                                                                                               
                        port_in_redirect off;                                                                                                  
                                                                                                                                               
                        set $balancer_ewma_score -1;                                                                                           
                        set $proxy_upstream_name "test-nginx-demo-80";                                                                         
                        set $proxy_host          $proxy_upstream_name;                                                                         
                        set $pass_access_scheme  $scheme;                                                                                      
                                                                                                                                               
                        set $pass_server_port    $server_port;                                                                                 
                                                                                                                                               
                        set $best_http_host      $http_host;                                                                                   
                        set $pass_port           $pass_server_port;                              
                                                                                                                                              
                        set $proxy_alternative_upstream_name "";                                                                               
                                                                                                                                               
                        client_max_body_size                    1m;                                                                            
                                                                                                                                               
                        proxy_set_header Host                   $best_http_host;                                                               
                                                                                                                                               
                        # Pass the extracted client certificate to the backend                                                                 
                                                                                                                                               
                        # Allow websocket connections                                                                                          
                        proxy_set_header                        Upgrade           $http_upgrade;                                               
                                                                                                                                               
                        proxy_set_header                        Connection        $connection_upgrade;                                         
                                                                                                                                               
                        proxy_set_header X-Request-ID           $req_id;                                                                       
                        proxy_set_header X-Real-IP              $remote_addr;                                                                  
                                                                                                                                               
                        proxy_set_header X-Forwarded-For        $remote_addr;                                                                  
                                                                                                                                               
                        proxy_set_header X-Forwarded-Host       $best_http_host;                                                               
                        proxy_set_header X-Forwarded-Port       $pass_port;                                                                    
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;                                                           
                        proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;                                                           
                                                                                                                                               
                        proxy_set_header X-Scheme               $pass_access_scheme;                                                           
                                                                                                                                               
                        # Pass the original X-Forwarded-For                                                                                    
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;                                                       
                                                                                                                                               
                        # mitigate HTTPoxy Vulnerability                                                                                       
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/                                          
                        proxy_set_header Proxy                  "";                                                                            
                                                                                                                                               
                        # Custom headers to proxied server                                                                                     
                                                                                                                                               
                        proxy_connect_timeout                   5s;                                                                            
                        proxy_send_timeout                      60s;                                                                           
                        proxy_read_timeout                      60s;                                                                           
                                                                                                                                               
                        proxy_buffering                         off;                                     
                        proxy_buffer_size                       4k;                                                                            
                        proxy_buffers                           4 4k;                                                                          
                                                                                                                                               
                        proxy_max_temp_file_size                1024m;                                                                         
                                                                                                                                               
                        proxy_request_buffering                 on;                                                                            
                        proxy_http_version                      1.1;                                                                           
                                                                                                                                               
                        proxy_cookie_domain                     off;                                                                           
                        proxy_cookie_path                       off;                                                                           
                                                                                                                                               
                        # In case of errors try the next upstream server before returning an error                                             
                        proxy_next_upstream                     error timeout;                                                                 
                        proxy_next_upstream_timeout             0;                                                                             
                        proxy_next_upstream_tries               3;                                                                             
                                                                                                                                               
                        # Custom Response Headers                                                                                              
                                                                                                                                               
                        proxy_pass http://upstream_balancer;                                                                                   
                                                                                                                                               
                        proxy_redirect                          off;                                                                           
                                                                                                                                               
                }                                                                                                                              
                                                                                                                                               
        }                                                                                                                                      
        ## end server www.test.com

longwuyuan · 2024-10-28T10:57:00Z

Its not a problem of the ingress-nginx controller

zzzzoy · 2024-10-28T12:13:12Z

what I'm puzzled is that the ingress attempts to associate with the pod through the service, but it does not perceive the modify of the service.
when the service is deleted, I'm sure it will affect the ingress.

zzzzoy added the kind/bug Categorizes issue or PR as related to a bug. label Oct 18, 2024

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Oct 18, 2024

k8s-ci-robot added the needs-priority label Oct 18, 2024

strongjz added this to [SIG Network] Ingress NGINX Oct 18, 2024

zzzzoy changed the title ~~when the service port changes, the ingress cannot detect the change and adjust the forwarding port.~~ when the service port changes, the ingress cannot detect the change and adjust the upstream port. Oct 18, 2024

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Oct 18, 2024

k8s-ci-robot closed this as completed Oct 18, 2024

github-project-automation bot moved this to Done in [SIG Network] Ingress NGINX Oct 18, 2024

k8s-ci-robot reopened this Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

when the service port changes, the ingress cannot detect the change and adjust the upstream port. #12209

when the service port changes, the ingress cannot detect the change and adjust the upstream port. #12209

zzzzoy commented Oct 18, 2024 •

edited

Loading

k8s-ci-robot commented Oct 18, 2024

longwuyuan commented Oct 18, 2024

k8s-ci-robot commented Oct 18, 2024

zzzzoy commented Oct 28, 2024

k8s-ci-robot commented Oct 28, 2024

longwuyuan commented Oct 28, 2024

zzzzoy commented Oct 28, 2024 •

edited

Loading

longwuyuan commented Oct 28, 2024

zzzzoy commented Oct 28, 2024 •

edited

Loading

when the service port changes, the ingress cannot detect the change and adjust the upstream port. #12209

when the service port changes, the ingress cannot detect the change and adjust the upstream port. #12209

Comments

zzzzoy commented Oct 18, 2024 • edited Loading

k8s-ci-robot commented Oct 18, 2024

longwuyuan commented Oct 18, 2024

k8s-ci-robot commented Oct 18, 2024

zzzzoy commented Oct 28, 2024

k8s-ci-robot commented Oct 28, 2024

longwuyuan commented Oct 28, 2024

zzzzoy commented Oct 28, 2024 • edited Loading

longwuyuan commented Oct 28, 2024

zzzzoy commented Oct 28, 2024 • edited Loading

zzzzoy commented Oct 18, 2024 •

edited

Loading

zzzzoy commented Oct 28, 2024 •

edited

Loading

zzzzoy commented Oct 28, 2024 •

edited

Loading