OpenProxy: Multi-Provider LLM Routing

Expert guidance for working with OpenProxy, a high-performance Rust-based proxy server that routes and load-balances requests between multiple LLM providers.

What This Skill Does

Helps you build, test, configure, and understand OpenProxy's architecture - a sophisticated proxy server written in Rust that intelligently routes requests to OpenAI, Gemini, Anthropic, and other LLM providers based on the Host header, with support for weighted load balancing, health checks, HTTP/2, WebSocket proxying, and hot reloading.

Core Capabilities

1. Build and Test Commands

```bash

Build the release binary

cargo build --release

Run all tests

cargo test

Run unit tests only

cargo test --lib

Run a specific test by name

cargo test test_name

Lint and format code

cargo clippy

cargo fmt

Start the server

./target/release/openproxy start -c config.yml

./target/release/openproxy start -c config.yml -p /var/run/openproxy.pid

```

2. End-to-End Test Suite

Navigate to the `e2e` directory and install Python dependencies:

```bash

cd e2e

pip install openai pydantic "httpx[http2]" websockets websocket-client pyyaml anthropic

```

Run individual test suites:

`python test_https.py` - HTTPS proxy functionality

`python test_http.py` - HTTP proxy functionality

`python test_websocket.py` - WebSocket proxying

`python test_host_path.py` - Host-based routing

`python test_anthropic_oauth.py` - Anthropic OAuth authentication

`python test_anthropic_api.py` - Anthropic API proxying

`python test_auth_selection.py` - Authentication selection logic

`python test_rewrite_auth_selection.py` - Auth rewriting and selection

`python test_connect_tunnel.py` - HTTP CONNECT tunneling

`python test_forward.py` - Forward proxy mode

`python test_h2_upstream.py` - HTTP/2 upstream connections

`python test_h2_h1_fallback.py` - HTTP/2 to HTTP/1.1 fallback

`python test_h2_large_body.py` - HTTP/2 large body handling

`python test_hot_upgrade.py` - Hot upgrade functionality

`python test_health_check_auth.py` - Health check with authentication

`python test_openai_realtime.py` - OpenAI Realtime API

Architecture Overview

OpenProxy routes requests to multiple LLM providers based on the `Host` header with intelligent load balancing and health monitoring.

Key Modules

**Configuration & Core State (`src/lib.rs`)**

Configuration loading from YAML

Provider selection with weighted load balancing

TLS setup and certificate management

Global `Program` state management

**Request Handling (`src/worker/mod.rs`)**

HTTP/1.1 and HTTP/2 request processing

WebSocket upgrade and proxying

`UpstreamInfo` struct for provider connection details

**Provider Implementations (`src/provider/mod.rs`)**

Provider trait defining authentication and header handling

OpenAI, Gemini, Anthropic, Forward provider implementations

Static and dynamic authentication (OAuth via `$(command)` pattern)

**HTTP Parsing (`src/http/mod.rs`)**

HTTP header parsing and manipulation

`Payload` struct with `next_block()` state machine for streaming

Header filtering and transformation logic

**Streaming Helpers (`src/http/reader/mod.rs`)**

`LimitedReader` for Content-Length-based streaming

`ChunkedReader`/`ChunkedWriter` for Transfer-Encoding: chunked

**Connection Management (`src/executor/mod.rs`)**

Connection pooling (`Pool<K, V>`)

Health check execution and provider availability tracking

**HTTP/2 Client (`src/h2client/mod.rs`)**

HTTP/2 upstream connections with TLS

Connection pooling for multiplexed streams

**WebSocket Support (`src/websocket/mod.rs`)**

WebSocket upgrade detection

Bidirectional proxying for WebSocket connections

Request Flow

1. **Client Connection**: TCP connection established (HTTP or HTTPS)

2. **TLS Handshake** (HTTPS): ALPN negotiation selects h2 or http/1.1

3. **Worker Spawned**: `Executor` creates a `Worker` to handle the connection

4. **Request Parsing**: Worker reads request, extracts `Host` header

5. **Provider Selection**: `Program::select_provider()` finds matching provider(s) by host

6. **Weighted Selection**: If multiple healthy providers match, weighted random selection is applied

7. **Request Proxying**: Worker proxies request to selected provider

8. **Response Streaming**: Response streamed back to client

9. **WebSocket Upgrade** (if applicable): Bidirectional tunnel established for WebSocket connections

Provider Trait Interface

The `Provider` trait defines how each LLM provider handles authentication and headers:

**Static Authentication**

`auth_header()` - Returns static authentication header value

`auth_header_key()` - Returns authentication header name

**Dynamic Authentication**

`uses_dynamic_auth()` - Whether provider supports dynamic auth

`get_dynamic_auth_header()` - Executes command to retrieve auth token (e.g., OAuth with `$(command)` pattern)

**Header Transformation**

`extra_headers()` - Additional headers to filter and transform

`transform_extra_header()` - Header value transformation logic (e.g., `anthropic-beta` for OAuth mode)

HTTP Header Processing

**HTTP/1.1 (`http/mod.rs`)**

`Payload` struct uses a state machine (`ReadState`) to stream request/response data

Headers are parsed and certain headers are filtered (`Host`, `Connection`, auth headers, extra headers)

`header_chunks: Vec<Range<usize>>` stores non-filtered header ranges

`split_header_chunks()` computes which parts of the header buffer to send upstream

Provider-specific headers are injected during streaming via `next_block()`

**HTTP/2 (`worker/mod.rs`)**

`UpstreamInfo` struct pre-computes transformed headers before sending upstream

`extra_header_keys` and `extra_headers_transformed` hold header transformations

Headers are filtered and replaced during request building in `proxy_h2()` and `proxy_h1()` (H1 fallback)

Protocol Support

**HTTP/1.1**: Plain HTTP and HTTPS with keep-alive

**HTTP/2**: Full multiplexing support, Extended CONNECT for WebSocket (RFC 8441), automatic H1 fallback for non-TLS upstreams

**HTTP CONNECT Tunnel**: Support for HTTP CONNECT method to establish tunnels (used by forward proxies)

**WebSocket**: Transparent proxying for both HTTP/1.1 upgrade and HTTP/2 CONNECT

Configuration

YAML-based configuration with hot-reload via SIGHUP signal.

**Required Fields**

`https_port` or `http_port` (at least one required)

`cert_file` and `private_key_file` (required for HTTPS)

`providers[]` array with at least one provider

**Provider Configuration**

`type`: openai, gemini, anthropic, or forward

`host`: Host header value for routing (e.g., `api.openai.com`)

`endpoint`: Actual backend endpoint URL

`api_key`: Static API key or dynamic command with `$(command)` syntax

`weight`: Load balancing weight (higher = more traffic)

`tls`: Enable TLS for upstream connection

`is_fallback`: Mark as fallback provider

**Optional Settings**

`https_bind_address` / `http_bind_address`: Bind address for listeners (default: 0.0.0.0)

`auth_keys`: Global authentication keys for client requests

`health_check.enabled`: Enable health checks (default: true)

`health_check.interval`: Health check interval in seconds (default: 60)

`http_max_header_size`: Maximum HTTP header size in bytes (default: 4096, min: 1024, max: 1MB)

Signal Handling

**SIGTERM/SIGINT**: Graceful shutdown (wait for in-flight requests)

**SIGHUP**: Reload configuration without restart

**SIGUSR2**: Hot upgrade (spawn new process with new config, then graceful shutdown)

Error Handling

Custom `Error` enum in `lib.rs` with variants:

`IO` - I/O errors

`TLS` - TLS/SSL errors

`H2` - HTTP/2 protocol errors

`HeaderTooLarge` - Request header exceeds `http_max_header_size`

`InvalidHeader` - Malformed HTTP headers

`NoProviderFound` - No matching provider for requested host

`DynamicAuthFailed` - OAuth or dynamic authentication command failed

Step-by-Step Instructions

When Modifying Code

1. **Understand the Module Structure**: Identify which module(s) your change affects (core state, worker, provider, HTTP parsing, etc.)

2. **Review the Provider Trait**: If adding a new provider or modifying authentication, check the `Provider` trait interface in `src/provider/mod.rs`

3. **Test Header Processing**: If changing header handling, understand the difference between HTTP/1.1 (`Payload` state machine) and HTTP/2 (`UpstreamInfo` pre-computation)

4. **Run Unit Tests**: After changes, run `cargo test --lib` to verify unit tests pass

5. **Run E2E Tests**: Navigate to `e2e/` and run relevant test suites to verify end-to-end behavior

6. **Check Linting**: Run `cargo clippy` and fix any warnings

7. **Format Code**: Run `cargo fmt` before committing

When Adding a New Provider

1. **Create Provider Implementation**: Add a new struct in `src/provider/mod.rs` implementing the `Provider` trait

2. **Define Authentication**: Implement `auth_header()` and `auth_header_key()` for static auth, or `uses_dynamic_auth()` and `get_dynamic_auth_header()` for OAuth

3. **Add Extra Headers**: If the provider requires special headers, implement `extra_headers()` and `transform_extra_header()`

4. **Update Configuration Enum**: Add the new provider type to the configuration parser

5. **Test**: Create an E2E test suite in `e2e/test_<provider>.py` following existing patterns

When Debugging Request Flow

1. **Enable Logging**: Run with `RUST_LOG=debug ./target/release/openproxy start -c config.yml`

2. **Trace Request Path**: Follow the request through:

- Worker receives connection (`worker/mod.rs`)

- Headers parsed (`http/mod.rs`)

- Provider selected (`lib.rs::select_provider()`)

- Request proxied (`worker/mod.rs::proxy_h1()` or `proxy_h2()`)

3. **Check Header Filtering**: Verify which headers are being filtered in `split_header_chunks()` (HTTP/1.1) or `UpstreamInfo` construction (HTTP/2)

4. **Monitor Health Checks**: If provider selection fails, check health check status in logs

When Configuring Production Deployment

1. **Create YAML Configuration**: Define providers with appropriate weights, endpoints, and API keys

2. **Enable Health Checks**: Set `health_check.enabled: true` and adjust interval as needed

3. **Configure TLS**: Provide valid `cert_file` and `private_key_file` for HTTPS

4. **Set Bind Addresses**: Use `https_bind_address` and `http_bind_address` to control network interfaces

5. **Test Configuration**: Run `./target/release/openproxy start -c config.yml` and verify startup logs

6. **Enable Hot Reload**: Send SIGHUP signal to reload configuration without downtime

7. **Plan for Upgrades**: Use SIGUSR2 for hot upgrades (new binary takes over, old process gracefully shuts down)

Constraints and Important Notes

**At least one port required**: Configuration must specify either `https_port` or `http_port`

**HTTPS requires certificates**: Both `cert_file` and `private_key_file` are mandatory for HTTPS

**Header size limits**: Default max header size is 4KB (configurable, max 1MB) to prevent memory exhaustion

**Health checks affect selection**: Unhealthy providers are excluded from weighted selection

**Dynamic auth caching**: OAuth tokens from `$(command)` are not cached - command runs on every request

**HTTP/2 requires TLS**: HTTP/2 support for upstream connections requires `tls: true` on provider

**WebSocket multiplexing**: HTTP/2 Extended CONNECT (RFC 8441) allows multiplexing WebSocket connections

**Connection pooling**: Both HTTP/1.1 and HTTP/2 connections are pooled for performance

**Graceful shutdown timeout**: In-flight requests have a grace period before forced termination

Example Usage

**Configuration file (config.yml)**:

```yaml

https_port: 8443

http_port: 8080

cert_file: /path/to/cert.pem

private_key_file: /path/to/key.pem

health_check:

enabled: true

interval: 60

providers:

- type: openai

host: api.openai.com

endpoint: https://api.openai.com

api_key: sk-...

weight: 3

tls: true

- type: anthropic

host: api.anthropic.com

endpoint: https://api.anthropic.com

api_key: $(aws secretsmanager get-secret-value --secret-id anthropic-key --query SecretString --output text)

weight: 2

tls: true

- type: gemini

host: generativelanguage.googleapis.com

endpoint: https://generativelanguage.googleapis.com

api_key: AIza...

weight: 1

tls: true

is_fallback: true

```

**Build and run**:

```bash

cargo build --release

./target/release/openproxy start -c config.yml

```

**Test with curl**:

```bash

Route to OpenAI (Host header determines provider)

curl -H "Host: api.openai.com" https://localhost:8443/v1/chat/completions

Route to Anthropic

curl -H "Host: api.anthropic.com" https://localhost:8443/v1/messages

```

**Hot reload configuration**:

```bash

Edit config.yml, then:

kill -HUP $(cat /var/run/openproxy.pid)

```

**Hot upgrade binary**:

```bash

Deploy new binary, then:

kill -USR2 $(cat /var/run/openproxy.pid)

```

OpenProxy: Multi-Provider LLM Routing

OpenProxy: Multi-Provider LLM Routing

What This Skill Does

Core Capabilities

1. Build and Test Commands

Build the release binary

Run all tests

Run unit tests only

Run a specific test by name

Lint and format code

Start the server

2. End-to-End Test Suite

Architecture Overview

Key Modules

Request Flow

Provider Trait Interface

HTTP Header Processing

Protocol Support

Configuration

Signal Handling

Error Handling

Step-by-Step Instructions

When Modifying Code

When Adding a New Provider

When Debugging Request Flow

When Configuring Production Deployment

Constraints and Important Notes

Example Usage

Route to OpenAI (Host header determines provider)

Route to Anthropic

Edit config.yml, then:

Deploy new binary, then:

Reviews (0)