[Design] Semantic Caching

[missBerg]

Produce a design proposal for the feature Semantic Caching.
Include:

• Motivation
• Feature Definition
• Control Plane API
• Technical Implementation Proposal

This issue was created from conversation during Dec 5th Community meeting
https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#bookmark=id.l5miyf5qkodx

[Krishanx92]

Motivation

Semantic caching in an LLM Gateway reduces API call costs by reusing responses for semantically similar requests, minimizing expensive LLM interactions. Traditional caching fails when queries have slight differences in wording, while semantic caching identifies intent using vector embeddings. This approach significantly improves response times, as cache hits are served in milliseconds compared to LLM processing delays. By reducing redundant LLM requests, semantic caching enhances cost efficiency and optimizes token usage. Ultimately, semantic caching achieves higher cache hit rates and smarter cache utilization, making the LLM Gateway more efficient.

Key Benefits

• Latency Reduction: Quicker response times since computation is bypassed.
• Cost Efficiency: Reduced computation costs for API-based LLMs.
• Improved Scalability: Handles higher throughput by avoiding redundant computations.

Will provide detail documentation for this.