x509: Certificate signed by unknown authority

[sachinshaji]

We use Envoy Gateway(installed using helm chart) as our Gateway API in MicroK8s cluster(redhat VM, air-gapped). We have our backend application and keycloak server running as pod inside micro K8s redhat VM. Our domain 'https://ppp.company.cloud' works as expected without using keycloak as our authentication server. As everything is in air-gapped installation I use firefox browser in my VM to access the 'https://ppp.company.cloud' domain(/etc/hosts entries are made) which has a self signed certificate.

But once we create a security policy I starts to get following error.
OIDC: error fetching endpoints from issuer: get "https://ppp.company.cloud/realms/myrealm/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority.

How can I make the security policy trust the self-signed certificate I have created. My firefox browser doesnot trust it but gives me warning. Is there any option to make the security policy by pass the certificate validation. Could you please help here.

Sharing our security policy yaml

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: eg-ppp-oidc-security-policy
  namespace: ppp-dev-ns
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: gateway-ppp-app-routes-oidc
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: all-gw-ppp-api-routes-oidc
  oidc:
    provider:
      issuer: "https://auth.ppp.company.cloud/realms/myrealm"
      authorizationEndpoint: "https://auth.ppp.company.cloud/realms/myrealm/protocol/openid-connect/auth"
      tokenEndpoint: "https://auth.ppp.company.cloud/realms/myrealm/protocol/openid-connect/token"
    clientID: "company.ppp.envoy-oidc.client"
    clientSecret:
      name: "company.ppp.envoy-oidc.client-secret"
    redirectURL: "https://ppp.company.cloud/pppapp/main/oauth2/callback"
    logoutPath: "/pppapp/main/logout"
    forwardAccessToken: true
    refreshToken: true
    cookieNames:
      accessToken: werwerwe
      idToken: ewrwewe

[arkodg]

the docs are not in yet, but you should be able to achieve this using the backendRefs field to link to a Backend and specify a custom ca cert using BackendTLSPolicy
there's a partial config in here https://github.com/envoyproxy/gateway/blob/main/internal/gatewayapi/testdata/securitypolicy-with-oidc-backendcluster.in.yaml

cc @zhaohuabing

[sachinshaji]

Thanks for the replay.
We follow the documentation you have mentioned. We have created- 'Backend' and 'BackendTLSPolicy' and use the 'backendRefs' inside security policy. But still we get the same certificate error.

Let me share the components we have created.

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: enable-backend-tls
  namespace: default
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: tls-backend
    sectionName: "443"
  validation:
    caCertificateRefs:
    - name: example-ca
      group: ''
      kind: ConfigMap
    hostname: www.example.com

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: enable-backend-tls
  namespace: default
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: tls-backend
    sectionName: "443"
  validation:
    caCertificateRefs:
    - name: example-ca
      group: ''
      kind: ConfigMap
    hostname: www.example.com

backends:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: Backend
  metadata:
    name: backend-fqdn
    namespace: envoy-gateway
  spec:
    endpoints:
    - fqdn:
        hostname: 'oauth.ppp.company.com'
        port: 443
securityPolicies:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    namespace: envoy-gateway
    name: policy-for-gateway
    uid: b8284d0f-de82-4c65-b204-96a0d3f258a1
  spec:
    targetRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: gateway-1
    oidc:
      provider:
        backendRefs:
        - group: gateway.envoyproxy.io
          kind: Backend
          name: backend-fqdn
          port: 443
        backendSettings:
          retry:
            numRetries: 3
            perRetry:
              backOff:
                baseInterval: 1s
                maxInterval: 5s
            retryOn:
              triggers: ["5xx", "gateway-error", "reset"]
        issuer: "https://oauth.ppp.company.com"
        authorizationEndpoint: "https://oauth.ppp.company.com/oauth2/v2/auth"
        tokenEndpoint: "https://oauth.ppp.company.com/token"
      clientID: "client1.apps.googleusercontent.com"
      clientSecret:
        name: "client1-secret"
      redirectURL: "https://www.example.com/bar/oauth2/callback"
      logoutPath: "/bar/logout"
      forwardAccessToken: true
      defaultTokenTTL: 30m
      refreshToken: true
      defaultRefreshTokenTTL: 24h

Could you please help us here. We have been trying this for last 2 days and could not get any resoution here. We have created the configmap with correct ca.crt also. Not able to debug this issue. Could you suggest any work around if possible?

[sachinshaji]

Similar discussion at #4838 may be its more useful

[sachinshaji]

May I ask one more question here, how is SecurityPolicy trying to connect to the issuers and verify the certificate?
AFAIK, SecurityPolicy is not creating any pod internally. Can you please help me understand is it actually connecting to any pod/deployment.
We install envoy using helm chart. As part of that a deployment(envoy-gateway) is created. Then when we create a Gateway K8's object and another deployment(envoy-proxy) is getting created.
To which deployment does the SecurityPolicy connects to? Is it to the envoy-gateway which is installed as part of helm or is it to the envoy-proxy which is installed with kubectl.
Can we fix this issue if we make any of these deployments/pods trust our self signed CA. Any help is really appreciated :)

[zhaohuabing]

Hi @sachinshaji You should define a BackendTLSPolicy targeting on the backend-fqdn Backend. Something like the follwing:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: enable-backend-tls
  namespace: default
spec:
  targetRefs:
  - group: ''
    kind: Backend
    name: backend-fqdn
    sectionName: "443"
  validation:
    caCertificateRefs:
    - name: example-ca
      group: ''
      kind: ConfigMap
    hostname: oauth.ppp.company.com