vLLM Mixbread Reranker Deployment

Deploy and test the `mixedbread-ai/mxbai-rerank-base-v2` reranker model using vLLM GPU server, ONNX CPU inference, or local Python wrapper. Supports Modal cloud deployment, Docker GPU server, and async benchmarking.

Overview

This skill guides you through deploying and testing a Mixbread Reranker with multiple deployment paths:

All commands use `uv run` for ad-hoc dependency injection without lock files.

Key Commands

Local Testing

Test the reranker locally using the Python wrapper:

```bash

Basic test with custom query and docs

make local-run QUERY="your query" DOCS=data/example_docs.json

Apple Silicon GPU (MPS)

make local-run-mps

CPU only

make local-run-cpu

Batch test multiple cases

make batch-run

```

The local runner uses the `mxbai_rerank` package and supports automatic device selection.

vLLM Docker (GPU Required)

Run vLLM reranker server in Docker with GPU acceleration:

```bash

Start vLLM server

make vllm-up-docker

Check server health

make vllm-health

Test /v1/rerank endpoint

make vllm-client

Verify Docker GPU access

make gpu-check

```

**Requirements**: NVIDIA GPU + NVIDIA Container Toolkit. Not supported on macOS/Windows.

ONNX CPU Deployment

Export model to ONNX format and apply int8 quantization:

```bash

Export to ONNX

uv run --with optimum[onnxruntime] --with transformers --with torch \

python scripts/export_onnx.py --model-id mixedbread-ai/mxbai-rerank-base-v2 --out-dir onnx/mxbai-base

Quantize to int8

uv run --with onnxruntime --with onnxruntime-tools \

python scripts/quantize_onnx.py --model-path onnx/mxbai-base/model.onnx --out-path onnx/mxbai-base/model-int8.onnx

Run local ONNX inference

uv run --with onnxruntime --with transformers \

python src/run_onnx.py --query "your query" --docs-file data/example_docs.json --model-dir onnx/mxbai-base --model-file model-int8.onnx --top-k 3

```

Modal Cloud Deployment

Deploy to Modal for serverless inference:

```bash

vLLM GPU deployment (dev server)

make modal-serve

Deploy to production

make modal-deploy

ONNX CPU deployment

make modal-serve-onnx

make modal-deploy-onnx

```

**Environment toggle**:

```bash

FAST_BOOT=true uvx modal serve modal_app.py

```

Benchmarking

Test reranker performance with BEIR datasets:

```bash

Build test dataset (scifact example)

make bench-build BENCH_DATASET=scifact BENCH_LIMIT=100

Run async benchmark

make bench-run BENCH_URL=https://<modal-url> CONCURRENCY=16 BENCH_METRICS=1

```

Architecture Details

Model Configuration

The Mixbread v2 reranker requires specific vLLM overrides:

```json

{

"architectures": ["Qwen2ForSequenceClassification"],

"classifier_from_token": ["0", "1"],

"method": "from_2_way_softmax"

}

```

Key Files

Modal Deployment Patterns

ONNX Implementation

Development Tips

Testing New Ideas

Use throwaway `uv run` one-liners before editing code:

```bash

uv run --with <libs> python -c "<code>"

```

For longer experiments, write to scratch files but don't commit clutter.

Dependency Management

All dependencies installed ad-hoc via `uv run --with <package>`. No requirements.txt. Modal apps pin versions in `.uv_pip_install()`. Update versions in `modal_app.py` and `modal_app_onnx.py` when upgrading.

Performance Tuning

GPU Requirements

Test Data Files

Constraints