ValidatingWebhookConfiguration for EnvoyProxy?

[Skaronator]

This is just a discussion because I'm not sure if this already somewhat exist and it's just a bug or just a missing feature.

Currently its possible to successfully deploy a EnvoyProxy config change without getting feedback that this config is broken and results in a non-working deployment.

For example when I deploy this EnvoyProxy config the kubectl apply command runs successfully:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: main-config
  namespace: network
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
              matchLabelKeys:
               - pod-template-hash

But looking in the logs of envoy-gateway I can see the following error:

2024-12-03T09:53:47.209Z    ERROR    watchable    message/watchutil.go:80    observed an error    {"runner": "infrastructure", "error": "failed to create or update deployment network/envoy-network-public-b0b37630: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-network-public-b0b37630  network    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:public gateway.envoyproxy.io/owning-gateway-namespace:network] map] ] ] ]},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: public,gateway.envoyproxy.io/owning-gateway-namespace: network,},MatchExpressions:]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:public gateway.envoyproxy.io/owning-gateway-namespace:network] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] ] ] ]} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-network-public-b0b37630,},Items:]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil nil}}] ] [{envoy docker.io/envoyproxy/envoy:distroless-dev [envoy] [--service-cluster network/public --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n  access_log:\n  - name: envoy.access_loggers.file\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n      path: /dev/null\n  address:\n    socket_address:\n      address: 127.0.0.1\n      port_value: 19000\nlayered_runtime:\n  layers:\n  - name: global_config\n    static_layer:\n      envoy.restart_features.use_eds_cache_for_ads: true\n      re2.max_program_size.error_level: 4294967295\n      re2.max_program_size.warn_level: 1000\ndynamic_resources:\n  ads_config:\n    api_type: DELTA_GRPC\n    transport_api_version: V3\n    grpc_services:\n    - envoy_grpc:\n        cluster_name: xds_cluster\n    set_node_on_first_message_only: true\n  lds_config:\n    ads: {}\n    resource_api_version: V3\n  cds_config:\n    ads: {}\n    resource_api_version: V3\nstatic_resources:\n  listeners:\n  - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n    address:\n      socket_address:\n        address: '0.0.0.0'\n        port_value: 19001\n        protocol: TCP\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: eg-ready-http\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: prometheus_stats\n              domains:\n              - \"*\"\n              routes:\n              - match:\n                  prefix: /stats/prometheus\n                route:\n                  cluster: prometheus_stats\n          http_filters:\n          - name: envoy.filters.http.health_check\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n              pass_through_mode: false\n              headers:\n              - name: \":path\"\n                string_match:\n                  exact: /ready\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n  clusters:\n  - name: prometheus_stats\n    connect_timeout: 0.250s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: prometheus_stats\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: 127.0.0.1\n                port_value: 19000\n  - connect_timeout: 10s\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway.network.svc.cluster.local\n                port_value: 18000\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options:\n            connection_keepalive:\n              interval: 30s\n              timeout: 5s\n    name: xds_cluster\n    type: STRICT_DNS\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: /sds/xds-certificate.json\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: /sds/xds-trusted-ca.json\n              resource_api_version: V3\n  - name: wasm_cluster\n    type: STRICT_DNS\n    connect_timeout: 10s\n    load_assignment:\n      cluster_name: wasm_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway\n                port_value: 18002\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options: {}\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: /sds/xds-certificate.json\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: /sds/xds-trusted-ca.json\n              resource_api_version: V3\noverload_manager:\n  refresh_interval: 0.25s\n  resource_monitors:\n  - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n      max_active_downstream_connections: 50000\n  - name: \"envoy.resource_monitors.fixed_heap\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig\n      max_heap_size_bytes: 858993459\n  actions:\n  - name: \"envoy.overload_actions.shrink_heap\"\n    triggers:\n    - name: \"envoy.resource_monitors.fixed_heap\"\n      threshold:\n        value: 0.95\n  - name: \"envoy.overload_actions.stop_accepting_requests\"\n    triggers:\n    - name: \"envoy.resource_monitors.fixed_heap\"\n      threshold:\n        value: 0.98\n --log-level warn --cpuset-threads --drain-strategy immediate --drain-time-s 60]  [{metrics 0 19001 TCP }] ] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] map[cpu:{{150 -3} {<nil>} 150m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] ]} ] <nil> [{certs true <nil> /certs  <nil> } {sds false <nil> /sds  <nil> }] ] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:5,SuccessThreshold:1,FailureThreshold:1,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:30,TerminationGracePeriodSeconds:nil,} &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:&Capabilities{Add:],Drop:[ALL],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*65532,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:*65532,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},AppArmorProfile:nil,} false false false} {shutdown-manager docker.io/envoyproxy/gateway-dev:latest [envoy-gateway] [envoy shutdown-manager]  ] ] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>}  BinarySI}] ]} ] <nil> ] ] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:30,TerminationGracePeriodSeconds:nil,} &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:&Capabilities{Add:],Drop:[ALL],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*65532,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:*65532,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},AppArmorProfile:nil,} false false false}] ] Always 0x4002543838 <nil> ClusterFirst map] envoy-network-public-b0b37630  0x40025435f7  false false false <nil> nil ]   nil default-scheduler ] ] joypaas-cluster-core <nil> nil ] <nil> <nil> <nil> map] [{1 kubernetes.io/hostname DoNotSchedule nil <nil> <nil> <nil> [pod-template-hash]}] <nil> nil <nil> ] ]}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-network-public-b0b37630\" is invalid: spec.template.spec.topologySpreadConstraints[0].matchLabelKeys: Forbidden: must not be specified when labelSelector is not set"}

Importent infos from the log above:

invalid: spec.template.spec.topologySpreadConstraints[0].matchLabelKeys: Forbidden: must not be specified when labelSelector is not set

I'd like to have this feedback while applying the CR change so that the broken config won't get applied and I actually get feedback of the deployment instead of looking in the logs later.

[arkodg]

#2717 should hopefully address this

[Skaronator]

Awesome!