Support custom backendRefs via extensions

[muwaqar]

Description:
Currently Envoy Gateway implements support for the following BackendReference.

• Service (support: Core)
• ServiceImport (support: Experimental)
• Backend (support: Envoy Gateway custom)

However, vendors may need to support specific backends (S3, EC2, Lambda, ...) which EG does not support natively. In that case, the ask is to enhance the Extensions sub-system to invoke a custom extension which can program the filter chain, route and upstream cluster for the configured backend correctly.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

I could find some previous discussion in #4423, specifically the comment at #4423 (comment)

Envoy Gateway extensions doc also mentions support for custom backendRefs is expected in future, but I could not find an issue tracking this, hence raising this.

Allow Extension Developers to introduce their own custom resources for extending the Gateway-API via ExtensionRefs, policyAttachments (future) and backendRefs (future).

[arkodg]

+1 @muwaqar, @guydc and @zhaohuabing have already +1'd this idea in #4423
would you be interested in driving this work ?

[arkodg]

reg AWS backends, there's this issue #688 tracking the work, although aws is a vendor specific backend, its fine to add a aws setting in the Backend resource since a Envoy filter exists for this in Envoy Proxy (we established this precedence when adding the datadog tracing field, which also is a native option in envoy proxy)

[zhaohuabing]

+1 native support for aws backend as they're widely used, starting with lambda could be a great first step.

[muwaqar]

great to hear there is broader consensus on this, is there a extension hook/api design sketched out or discussed for this already?

[zhaohuabing]

Native support for aws lambda can be added to https://github.com/envoyproxy/gateway/blob/main/api/v1alpha1/backend_types.go and implemented in the Gateway API translator and xDS translator.

For custom Backend, there's no extension hook/api design in plance, and this will require furtuer discussion.

[arkodg]

Here are some links related to the extension manager route

• Support for custom policies feat: Adding extension server policy handling. #3371
• Original Design Document https://github.com/envoyproxy/gateway/blob/main/site/content/en/contributions/design/extending-envoy-gateway.md
  cc @liorokman

[muwaqar]

I was reading through the Envoy AI Gateway proposal document and seems like they have a similar use-case to set custom resources as HTTPRoute backendRefs (pasting snippet from the doc below).

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
   name:  mistral-routing
spec:
   rules:
   - matches:
     - path:
        type: prefix
        value: /mistral
     backendRefs:
     - group: serving.kserve.io/v1alpha1
       kind: LLMBackend
       name: kserve-mistral

is this enhancement to envoy-gateway already scoped and tracked by that effort?

[arkodg]

imo its unrelated, afaik the updated API is envoyproxy/ai-gateway#20 and the custom backend links to a custom route type and the ai-gateway controller converts those custom types into Envoy Gateway resources (like HTTPRoute and Backend , (@mathetake please keep me honest here)

[mathetake]

yeah we need the custom HTTPRoute-ish stuff as well not only Backend, so this won't help us much (by the way the design doc you shared is obsolete and not necessarily reflect the actual stuff Bloomberg and Tetrate worked on)

[liorokman]

Custom backends can already be added with the extension server, but it's a bit complicated.

The idea is to add a regular route with one of the standard backends, and use the VirtualHost modification hook to modify the standard backend's translation into whatever you need.

We could make this easier to do with the extension server by providing a "PlaceholderBackend" that Envoy Gateway doesn't actually know how to translate to XDS, but can send to an extension server for further handling.

For example:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: PlaceholderBackend
metadata:
  name: go-to-s3
spec: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: gateway.envoyproxy.io
          kind: PlaceholderBackend
          name: go-to-s3
      matches:
        - path:
            type: PathPrefix
            value: /

When the Extension Server receives the call to the VirtualHost hook for the XDS generated for the go-to-s3 backend, it can fill in the details in XDS as needed.