Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Duplicated listeners after federating a zone #9006

Closed
slonka opened this issue Jan 23, 2024 · 8 comments · Fixed by #9542
Closed

Duplicated listeners after federating a zone #9006

slonka opened this issue Jan 23, 2024 · 8 comments · Fixed by #9542
Assignees
@jijiechen
Labels
kind/bug A bug triage/needs-reproducing Someone else should try to reproduce this

Comments

@slonka
Copy link
Contributor

slonka commented Jan 23, 2024

What happened?

got:

 2024-01-23T09:59:44.403Z    INFO    xds.nack-backoff    config was previously rejected by Envoy. Applying backoff before resending it    {"backoff": "5s", "nodeID": "default.demo-app-c7cd6588b-wxn6t.kuma-demo", "reason": "Error adding/updating listener(s) outbound:2 │
│ 40.0.0.1:80: error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener\noutbound:240.0.0.0:80: error adding listener: 'outbound:240.0.0.0:80' has duplicate address '240.0.0.0:80' as existing listener\n"}                 │

after federating a zone with demo installed, @bartsmykla mentioned that this could be related to #6717 - that's why I assigned @jijiechen

Steps to reproduce:

  1. deploy a zone
  2. install demo
  3. deploy a global
  4. export resources using kumactl export
  5. federate zone (kumactl install with kds global address)
  6. check logs
@slonka slonka added triage/pending This issue will be looked at on the next triage meeting kind/bug A bug labels Jan 23, 2024
@jijiechen
Copy link
Member

Reproduced once, but not too much useful information captured.

This issue is a transitional state during the importing step of the federation:

  • the error occured on data planes (i.e. sidecars)
  • the error indicates the data plane has received invalid LDS from a control plane hence it was rejected
  • the error went away quickly in a short period of time
  • traffic on the data plane is not impacted (the VIPs/.mesh domains are able to be accessed after the federation)
complete data plane logs 
2024-01-25T08:30:22.010Z	INFO	config	skipping reading config from file
2024-01-25T08:30:22.010Z	INFO	kuma-dp.run	effective configuration	{"config": "{\"controlPlane\":{\"caCert\":\"-----BEGIN CERTIFICATE-----\\nMIIDEDCCAfigAwIBAgIRAKwURu51rSVmOkAAM9tHfqQwDQYJKoZIhvcNAQELBQAw\\nEjEQMA4GA1UEAxMHa3VtYS1jYTAeFw0yNDAxMjUwODI4MzdaFw0zNDAxMjIwODI4\\nMzdaMBIxEDAOBgNVBAMTB2t1bWEtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\\nggEKAoIBAQDApuKyYXlcNrrzIMzamWnQahJxYyycQlw43Z8erkmi1kUqqkEWQzfY\\nAmQdHMMk0YUZacCFLcjMyY5VqOkn1zUWJchSNgTYwEIkLyN7HjobwdQ1V4B6SY0E\\n9geGr5pJLoLcULuV5jgAqlgQU0pEDvT6SZ3DaaCS+YxBgZpa1WKujLVMsvz/rIbC\\nVP5UxTd0Pmmfa+h9JCjNzdZJ2FJNPfqwgpIGtiuIaJG0sX/mQgfCnthKHAFxGLf3\\n20awgs7uP+VtW7yG6+Wtzu1ydr+vz9BXtEuL0oBSjH9ytlZ/1hj9qbGSh9xv7681\\nFo7+dkcIc7o3zH8S7rDRLBQKBNXS21zrAgMBAAGjYTBfMA4GA1UdDwEB/wQEAwIC\\npDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB\\n/zAdBgNVHQ4EFgQUftDh2kgss6k5SsXZzgFnseod48YwDQYJKoZIhvcNAQELBQAD\\nggEBAKYjBUeg7Jng8Q+h0kw2zOcu5IBM7+0F9u49qkcYq0yCQYewCHrWbQwf4KMD\\nlGc4qnsnhI9yvoLe6p4k47QdxP+TUo7vB1DibW1anPQwn30a6IWZNMw5rFWpz9gM\\n+3iiUAB/aywRUVBk4tiu2UxsGbZBBhS8TfayYVcFpcrFHhEMKRk1ufJs53xLxgrg\\nbWm2kikWRzIf3LrXQmWxIysHYunzqmeS5b2Hv3oIZHe8XLJUc7GbpByAB9agRQIU\\nEfoSv5UERYqHSp44Rhi4kSxasUEQXxUQBS0UXH+472ASKfZqU9MEtnn8aM1qP/KN\\ni+9GQvrbEx7TRk3DuS78l8tdwP4=\\n-----END CERTIFICATE-----\\n\",\"caCertFile\":\"\",\"retry\":{\"backoff\":\"3s\",\"maxDuration\":\"5m0s\"},\"url\":\"https://kong-mesh-control-plane.kong-mesh-system:5678\"},\"dataplane\":{\"drainTime\":\"30s\",\"mesh\":\"default\",\"name\":\"demo-app-d789d49c-j86js.kuma-demo\",\"proxyType\":\"dataplane\"},\"dataplaneRuntime\":{\"binaryPath\":\"envoy\",\"concurrency\":2,\"dataplaneTokenPath\":\"/var/run/secrets/kubernetes.io/serviceaccount/token\",\"metrics\":{},\"resources\":{}},\"dns\":{\"coreDnsBinaryPath\":\"coredns\",\"coreDnsEmptyPort\":15054,\"coreDnsPort\":15053,\"enabled\":true,\"envoyDnsPort\":15055,\"prometheusPort\":19153}}"}
2024-01-25T08:30:22.010Z	INFO	kuma-dp.run	generated configurations will be stored in a temporary directory	{"dir": "/tmp/kuma-dp-3804747063"}
2024-01-25T08:30:22.010Z	INFO	config	skipping reading config from file
2024-01-25T08:30:22.010Z	INFO	effective Kong Mesh DP configuration	{"config": "{\"auth\":{\"type\":\"\"},\"opa\":{\"addr\":\"localhost:8181\",\"configOverrides\":[],\"configPath\":\"\",\"diagnosticAddr\":\"0.0.0.0:8282\",\"enabled\":true,\"extAuthzAddr\":\"localhost:9191\"}}"}
2024-01-25T08:30:22.034Z	INFO	kuma-dp.run	fetched Envoy version	{"version": {"Build":"b5ca88acee3453c9459474b8f22215796eff4dde/1.28.0/Modified/RELEASE/BoringSSL","Version":"1.28.0","KumaDpCompatible":true}}
2024-01-25T08:30:22.034Z	INFO	kuma-dp.run	generating bootstrap configuration
2024-01-25T08:30:22.034Z	INFO	dataplane	trying to fetch bootstrap configuration from the Control Plane
2024-01-25T08:30:22.049Z	INFO	kuma-dp.run	received bootstrap configuration	{"adminPort": 9901}
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run	starting Kuma DP	{"version": "2.5.1"}
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run.access-log-streamer	starting resilient component ...
2024-01-25T08:30:22.051Z	INFO	access-log-streamer	cleaning existing access log pipe	{"file": "/tmp/kuma-dp-3804747063/kuma-al-demo-app-d789d49c-j86js.kuma-demo-default.sock"}
2024-01-25T08:30:22.051Z	INFO	access-log-streamer	creating access log pipe	{"file": "/tmp/kuma-dp-3804747063/kuma-al-demo-app-d789d49c-j86js.kuma-demo-default.sock"}
2024-01-25T08:30:22.051Z	INFO	starting resilient component ...
2024-01-25T08:30:22.051Z	INFO	opa	starting OPADS stream to receive OPA Config
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run.envoy	bootstrap configuration saved to a file	{"file": "/tmp/kuma-dp-3804747063/bootstrap.yaml"}
2024-01-25T08:30:22.051Z	INFO	metrics-hijacker	starting Metrics Hijacker Server	{"socketPath": "unix:///tmp/kuma-dp-3804747063/kuma-mh-demo-app-d789d49c-j86js.kuma-demo-default.sock"}
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run.envoy	starting Envoy	{"path": "/usr/bin/envoy", "arguments": ["--config-path", "/tmp/kuma-dp-3804747063/bootstrap.yaml", "--drain-time-s", "30", "--disable-hot-restart", "--log-level", "info", "--concurrency", "2"]}
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run.dns-server	configuration saved to a file	{"file": "/tmp/kuma-dp-3804747063/Corefile"}
2024-01-25T08:30:22.051Z	INFO	kuma-dp.run.dns-server	starting DNS Server (coredns)	{"args": ["-conf", "/tmp/kuma-dp-3804747063/Corefile", "-quiet"]}
2024-01-25T08:30:22.051Z	INFO	opa	waiting for OPA Config
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:413] initializing epoch 0 (base id=0, hot restart version=disabled)
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:415] statically linked extensions:
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.geoip_providers: envoy.geoip_providers.maxmind
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.connection_handler: envoy.connection_handler.default
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.regex_engines: envoy.regex_engines.google_re2
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.common.key_value: envoy.key_value.file_based
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   network.connection.client: default, envoy_internal
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.header_mutation, envoy.filters.http.upstream_codec
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.resolvers: envoy.ip
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   quic.http_server_connection: quic.http_server_connection.default
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.protocols: dubbo
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.request_id: envoy.request_id.uuid
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-01-25 08:30:22.069][26][info][main] [source/server/server.cc:471] HTTP header map info:
[2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474]   request header map: 680 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474]   request trailer map: 128 bytes: 
[2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474]   response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474]   response trailer map: 152 bytes: grpc-message,grpc-status
[2024-01-25 08:30:22.078][26][info][main] [source/server/server.cc:837] runtime: layers:
  - name: kuma
    static_layer:
      re2.max_program_size.warn_level: 1000
      re2.max_program_size.error_level: 4294967295
[2024-01-25 08:30:22.079][26][info][admin] [source/server/admin/admin.cc:68] admin address: 127.0.0.1:9901
[2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:159] loading tracing configuration
[2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:118] loading 1 static secret(s)
[2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:124] loading 2 cluster(s)
[2024-01-25 08:30:22.088][26][info][config] [source/server/configuration_impl.cc:128] loading 0 listener(s)
[2024-01-25 08:30:22.088][26][info][config] [source/server/configuration_impl.cc:145] loading stats configuration
[2024-01-25 08:30:22.089][26][warning][main] [source/server/server.cc:901] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-01-25 08:30:22.089][26][info][main] [source/server/server.cc:942] starting main dispatch loop
[2024-01-25 08:30:22.090][26][info][runtime] [source/common/runtime/runtime_impl.cc:579] RTDS has finished initialization
[2024-01-25 08:30:22.090][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
2024-01-25T08:30:23.062Z	INFO	opa	received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane
2024-01-25T08:30:23.062Z	INFO	opa	waiting for OPA Config
[2024-01-25 08:30:23.674][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:30:23.713][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 8 cluster(s), skipped 0 unmodified cluster(s)
[2024-01-25 08:30:23.713][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:200] cm init: initializing secondary clusters
[2024-01-25 08:30:23.715][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2024-01-25 08:30:23.715][26][info][main] [source/server/server.cc:923] all clusters initialized. initializing init manager
[2024-01-25 08:30:23.721][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000'
[2024-01-25 08:30:23.722][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:passthrough:ipv6'
[2024-01-25 08:30:23.723][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:dns'
[2024-01-25 08:30:23.724][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:30:23.724][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80'
[2024-01-25 08:30:23.725][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:passthrough:ipv6'
[2024-01-25 08:30:23.729][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'probe:listener'
[2024-01-25 08:30:23.729][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:passthrough:ipv4'
[2024-01-25 08:30:23.738][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin'
[2024-01-25 08:30:23.739][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379'
[2024-01-25 08:30:23.739][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.1:80'
[2024-01-25 08:30:23.740][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:passthrough:ipv4'
[2024-01-25 08:30:23.740][26][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:923] all dependencies initialized. starting workers
[2024-01-25 08:30:34.587][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:30:34.607][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 1 cluster(s), skipped 7 unmodified cluster(s)
[2024-01-25 08:30:34.613][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:30:34.619][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80'
[2024-01-25 08:37:52.213][26][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:152] StreamAggregatedResources gRPC config stream to ads_cluster closed: 13, 
2024-01-25T08:37:52.216Z	INFO	opa	closing OPADS stream
2024-01-25T08:37:52.216Z	ERROR	component terminated with an error	{"generationID": 1, "error": "could not receive OPA Config: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR", "errorVerbose": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR\ncould not receive OPA Config\ngithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa.(*opaConfigurer).Start\n\tgithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa/configurer.go:65\ngithub.com/kumahq/kuma/pkg/core/runtime/component.(*resilientComponent).Start.func1\n\tgithub.com/kumahq/[email protected]/pkg/core/runtime/component/resilient.go:43\nruntime.goexit\n\truntime/asm_arm64.s:1197"}
[2024-01-25 08:37:53.744][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:37:53.766][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 5 cluster(s), skipped 3 unmodified cluster(s)
[2024-01-25 08:37:53.767][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.0:80'
[2024-01-25 08:37:53.767][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.1:80'
[2024-01-25 08:37:53.768][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:dns'
[2024-01-25 08:37:53.775][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:37:53.780][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin'
[2024-01-25 08:37:53.781][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379'
2024-01-25T08:37:57.219Z	INFO	opa	starting OPADS stream to receive OPA Config
2024-01-25T08:37:57.236Z	INFO	opa	waiting for OPA Config
2024-01-25T08:37:58.239Z	INFO	opa	received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane
2024-01-25T08:37:58.239Z	INFO	opa	waiting for OPA Config
[2024-01-25 08:38:09.721][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000'
[2024-01-25 08:38:09.730][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:43:12.103][26][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:152] StreamAggregatedResources gRPC config stream to ads_cluster closed: 13, 
2024-01-25T08:43:12.107Z	INFO	opa	closing OPADS stream
2024-01-25T08:43:12.109Z	ERROR	component terminated with an error	{"generationID": 2, "error": "could not receive OPA Config: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR", "errorVerbose": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR\ncould not receive OPA Config\ngithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa.(*opaConfigurer).Start\n\tgithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa/configurer.go:65\ngithub.com/kumahq/kuma/pkg/core/runtime/component.(*resilientComponent).Start.func1\n\tgithub.com/kumahq/[email protected]/pkg/core/runtime/component/resilient.go:43\nruntime.goexit\n\truntime/asm_arm64.s:1197"}
[2024-01-25 08:43:13.323][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:43:13.323][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 0 cluster(s), skipped 8 unmodified cluster(s)
[2024-01-25 08:43:13.330][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin'
2024-01-25T08:43:17.112Z	INFO	opa	starting OPADS stream to receive OPA Config
2024-01-25T08:43:17.129Z	INFO	opa	waiting for OPA Config
2024-01-25T08:43:18.132Z	INFO	opa	received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane
2024-01-25T08:43:18.132Z	INFO	opa	waiting for OPA Config
[2024-01-25 08:43:31.174][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:43:31.188][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 3 cluster(s), skipped 5 unmodified cluster(s)
[2024-01-25 08:43:31.194][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:43:31.200][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80'
[2024-01-25 08:43:31.201][26][warning][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:1113] error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener
[2024-01-25 08:43:31.205][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000'
[2024-01-25 08:43:31.207][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379'
[2024-01-25 08:43:31.207][26][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) outbound:240.0.0.1:80: error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener

[2024-01-25 08:43:36.214][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.1:80'
[2024-01-25 08:43:36.214][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s)
[2024-01-25 08:43:36.229][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 2 cluster(s), skipped 6 unmodified cluster(s)
[2024-01-25 08:45:53.162][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.1:80'
[2024-01-25 08:45:53.162][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.0:80'
[2024-01-25 08:45:53.169][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'
[2024-01-25 08:45:53.171][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379'
[2024-01-25 08:45:54.185][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000'

@jijiechen
Copy link
Member

jijiechen commented Jan 25, 2024 •
edited
Loading

There are no unusual logs generated in any control planes:

  • original 2.5.x
  • upgraded 2.6.0-preview
  • federated 2.6.0-preview

All of logs generated from above pods are regular entries.

@jakubdyszkiewicz
Copy link
Contributor

Triage: we could double-check this with upgrade 2.5.0 to 2.6.0 without any federation. Increase the number of listeners (by adding more services) to get a better chance of hitting this issue.

@jakubdyszkiewicz jakubdyszkiewicz added triage/accepted The issue was reviewed and is complete enough to start working on it and removed triage/pending This issue will be looked at on the next triage meeting labels Jan 29, 2024
@ttreptow ttreptow mentioned this issue Feb 8, 2024
Closed
@ttreptow
Copy link
Contributor

ttreptow commented Feb 8, 2024

In my case, the problem does not go away. Some things I've noticed:

  • The duplicate listener is always for a service in the local zone
  • It's possible to use the kuma.io/transparent-proxying-reachable-services to avoid the error if the pod only needs services from the other zone
  • when I do a config dump on one of the affected data planes, the duplicate listener blocks are completely identical (i.e. the exact same block shows up twice in the type.googleapis.com/envoy.admin.v3.ListenersConfigDump section)

Regarding that last point, perhaps simply filtering out duplicate configs before sending them could be a quick fix?

@lahabana
Copy link
Contributor

@ttreptow does this still happen in 2.6.1 ? Seems like there were multiple things at play at the same time here.

@lahabana lahabana added triage/needs-information Reviewed and some extra information was asked to the reporter and removed triage/accepted The issue was reviewed and is complete enough to start working on it labels Feb 27, 2024
@ttreptow
Copy link
Contributor

I didn't see the issue this time when I upgraded from 2.5.2 to 2.6.1

@ttreptow
Copy link
Contributor

I spoke too soon actually, I do still see the issue

@jakubdyszkiewicz jakubdyszkiewicz added triage/needs-reproducing Someone else should try to reproduce this and removed triage/needs-information Reviewed and some extra information was asked to the reporter labels Mar 4, 2024
@jijiechen
Copy link
Member

@ttreptow Thanks for reporting.

Just wanted to confirm a few items. Do you have any headless serviecs in your mesh? On the services of the duplicated listeners, are there any properties different or interesting? Does this problem cause any actual traffic issue in your mesh? We are seeing this occurring on headless services, and this will be fixed in a coming patch release.

If headless services are not in your case and you don't mind, could you send us a copy of your dumpped envoy config? (Please remove any sensetive information like secrets, PII, etc). You may post it on this thread if you'd like, or you can also send it via email to [email protected]. Thank you very much.

@jijiechen jijiechen self-assigned this Mar 8, 2024
@jijiechen jijiechen mentioned this issue Mar 8, 2024
Closed
5 tasks
@jakubdyszkiewicz jakubdyszkiewicz mentioned this issue Mar 8, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jijiechen jijiechen

Labels
kind/bug A bug triage/needs-reproducing Someone else should try to reproduce this
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(xds): duplicated listeners jakubdyszkiewicz/kuma
5 participants
@lahabana @jijiechen @slonka @ttreptow @jakubdyszkiewicz