# Search Result Reranking

MiniSearch optionally reranks search results using a cross-encoder model running in-process via ONNX Runtime. This secondary search stage reorders initial SearXNG results based on their semantic relevance to the user's query.

## Architecture Overview

The reranking subsystem consists of four components:

| Component | File | Responsibility |
|-----------|------|----------------|
| Service Manager | `server/rerankerService.ts` | Readiness state, reranking inference |
| Model Loading | `server/utils/onnxModelLoader.ts` | Shared model download, tokenizer loading, and session creation (used by both the reranker and the bi-encoder) |
| Ranking Logic | `server/rankSearchResults.ts` | Score-based filtering and result reordering |
| Server Hook | `server/rerankerServiceHook.ts` | Startup/shutdown coordination with Vite server |

## Service Lifecycle

### Startup

The `rerankerServiceHook` starts the reranker during server initialization:

1. Downloads the model and tokenizer from HuggingFace if not present (`cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`)
2. Creates an ONNX Runtime inference session
3. Performs a warmup inference (`query: "test"`, `documents: ["test document"]`) to ensure the graph is initialized
4. Sets `isReady = true`

There is no child process, port, or health endpoint: inference runs inside the Node process.

### Model Cache Integrity

`downloadFileFromHuggingFaceRepository` compares each cached file against the size the Hub reports, so a file downloaded only in part is replaced instead of being loaded. Without that check a truncated model made every later startup fail with `Protobuf parsing failed`, and only deleting `server/models/` by hand recovered.

Downloads land on a sibling `.part-<pid>` path and are renamed once the whole body is on disk, which keeps an interrupted write from being visible where the next startup would trust it. Files are also checked individually, so a missing tokenizer is fetched on its own while the 119MB model stays cached.

The size check costs one metadata request per file at startup (roughly 600ms for the three of them). When the Hub is unreachable the cached files are used as they are, so an offline server still starts with a warm cache.

### Execution Providers

The session is created with `["cpu"]`, with no configuration to set, and there is no second attempt to fall back from. This decision lives in `server/utils/onnxModelLoader.ts` and governs both the reranker and the bi-encoder.

The reranker ships as a dynamically quantized graph, and the WebGPU provider has no kernels for its integer matmuls. It registers happily and then hands every one of them back to the CPU, paying a round trip each time: 812ms against 172ms of the same work, with scores drifting by as much as 1.15 and reordering results. The bi-encoder is fp32 and was not benchmarked on accelerators, but inherits the CPU-only choice.

| Provider | Availability in the Node binding | Notes |
|----------|----------------------------------|-------|
| `cpu` | Everywhere | The only provider used |
| `webgpu` | Windows, Linux x64, macOS | Not used: no integer-matmul kernels, so a quantized graph runs ~4.7x slower than on CPU and its scores drift |
| `cuda` | Linux x64 (CUDA v12) | Not used: the binaries are not bundled, and would need `npm install onnxruntime-node --onnxruntime-node-install=cuda12` |
| `coreml` | macOS | Not used: slower than CPU for this model's dynamic shapes |

### One Document per Call

Documents are scored one at a time rather than in batches, which is what keeps a score a property of its own `(query, document)` pair.

Dynamic quantization computes each activation tensor's scale from that tensor's own range at runtime. Padding rows out to a shared width puts the pad positions inside that range, and while the attention mask keeps them out of attention it cannot keep them out of the scale, so the quantization of the real tokens shifts. Batches of 10 moved logits by up to 1.29 at ordinary snippet lengths and reordered 2 of 10 fixtures depending only on which documents shared a batch. The fp32 export of the same model shows a difference of exactly zero, so this is a property of quantization and not of the graph.

Scoring one pair at a time removes the padding, and with it the coupling. It also suits the reason batching was introduced in the first place: `onnxruntime-node` wraps a synchronous native call, and one pair holds the event loop for about 13ms instead of the 50ms a batch of 10 takes. The cost is throughput, roughly 12% more wall time for 30 documents on two threads and up to 47% more where there are cores to spare.

### Shutdown

On server close, `stopRerankerService()` clears the readiness flag and releases the inference session.

## Health Monitoring

`getRerankerStatus()` reports whether the model finished loading. The search endpoint checks it before attempting ranking and falls back to unranked SearXNG results if the reranker is unavailable.

## Reranking Process

### Document Preparation

Each result is formatted as `` `${title}\n${snippet}` ``: the cased title and snippet on either side of a newline, with no URL. The query keeps its original casing too. Unicode surrogates in both are sanitized before tokenization.

Documents are sent to the reranker whole. Truncation happens by tokens inside the reranker, not by characters here: `score()` caps each encoded `(query, document)` pair at `MAX_SEQUENCE_LENGTH` tokens, dropping tokens from the end of the document only, so the query and the trailing separator are preserved.

`MAX_SEQUENCE_LENGTH` is 512 because the model has 514 learned position embeddings and XLM-RoBERTa reserves two of them. It is a limit, not a preference: a 513-token pair fails outright with `indices element out of data bounds` at the position-embedding gather. Web snippets land around 33-63 tokens per pair, so truncation is rare in practice.

### Unicode Sanitization

`sanitizeUnicodeSurrogates()` validates Unicode surrogate pairs in input strings. Invalid surrogates are replaced with the Unicode replacement character (`�`). This prevents failures when processing malformed UTF-8 from web search results.

### Scoring and Filtering

The reranker returns the classifier's raw relevance logit for each document. Scores are deliberately not passed through a sigmoid, because the filter below is calibrated against the raw scale. Results are then filtered using a two-stage statistical approach:

1. **Score Normalization**: Scores are shifted to positive range by adding the absolute value of the minimum score
2. **Standard Deviation Filter**: Results below `mean - kStandardDeviationFactor * standardDeviation` are filtered out
   - `kStandardDeviationFactor = 0.3`
3. **Percentage Fallback**: If fewer than 40% of results pass the standard deviation filter, a fallback threshold is applied:
   - `minPercentageFallback = 0.4` (40% of the highest normalized score)

This filter is invariant to linear rescaling of the scores, since both the scores and the threshold scale together.

### Preserve Top Results Mode

When `preserveTopResults = true`, the ranking algorithm:

1. Keeps the original top result (first from SearXNG) at position 1
2. Filters remaining results by score
3. Takes up to 9 next-best results, sorted by reranker score
4. Appends remaining filtered results

This mode ensures the original top result is never displaced by the reranker, while still improving the ordering of subsequent results.

## Integration with Search Pipeline

The search endpoint coordinates reranking in `searchEndpointServerHook.ts`:

```
1. fetchSearXNG(query, searchType, limit) - internally fetches from SearXNG, deduplicates, and cleans results
2. Check getRerankerStatus()
   - If healthy: rankSearchResults(query, searchType, results, preserveTopResults)
   - If unhealthy: return unranked results (fallback)
3. Return structured JSON response
```

Reranking is applied to both text and image search results. For image results, titles and URLs are reformatted into the same text-tuple shape used for text search before being sent through the reranker, then the original image data (including thumbnails) is re-matched to the reranked order by URL. Text search passes `preserveTopResults: true` to keep the original top SearXNG result pinned, while image search does not.

## Model Details

| Property | Value |
|----------|-------|
| Model | mmarco-mMiniLMv2-L12-H384-v1 |
| Format | ONNX, dynamically quantized (`onnx/model_quint8_avx2.onnx`, 119MB) |
| HuggingFace Repo | cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 |
| Type | Cross-encoder reranker (XLM-RoBERTa, one relevance logit) |
| Size | 12 layers, 118M parameters, of which 96M are the 250k-token vocabulary |
| Max sequence length | 512 tokens, the model's own limit, enforced via `MAX_SEQUENCE_LENGTH` |
| Storage | `server/models/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/` |

It is trained on mMARCO, the machine-translated multilingual MS MARCO, and is genuinely multilingual: a 250k-token sentencepiece vocabulary shared across languages, which is also where most of the parameter count goes.

The model it replaced, `jina-reranker-v1-tiny-en`, was chosen on the belief that an English model ranked other languages well enough. Measured against 240 MIRACL queries across English, Spanish, French, German, Russian and Japanese, it scored 0.4636 nDCG@10, below the 0.4963 of the un-reranked first-stage order: on non-English content it was not adding ranking signal. This model scores 0.7992 on the same set. Part of the mechanism is the tokenizer, since an English WordPiece vocabulary spends 2.17x as many tokens on Russian and about 1.4x on Spanish, German and Japanese, so the English model paid more compute to see worse-fragmented subwords.

### Choosing the Quantized Export

The repository ships one build per CPU kernel family from the same weights: `qint8_arm64`, `qint8_avx512`, `qint8_avx512_vnni` and `quint8_avx2`. `quint8_avx2` is the one used, because unsigned activations sidestep the signed-int8 saturation that x64 without VNNI otherwise has to work around, and it measured no slower than the arm64 build when run on arm64. The `qint8_arm64` and `qint8_avx512` builds are bit-identical in output and score 0.8039, marginally above `quint8_avx2`, but that margin is inside the noise of a 240-query set and does not buy predictable behaviour on hosts whose instruction set cannot be known in advance.

Quantization costs nothing measurable here: 0.7992 against 0.7973 nDCG@10 for the fp32 export, for a quarter of the download and half the latency. That is a different conclusion from the one that held for the previous 33M-parameter model, where the `q8` export did measurably degrade ranking; the loss does not carry over to a model with 12 layers and a large embedding table.

The `O4` (fp16) export loads without complaint but is slower than fp32 on CPU, so it is not used either.

## Testing

`server/rerankerService.test.ts` runs in the default suite and covers Unicode sanitization, token truncation, and that every document is sent as its own unpadded row, with the ONNX Runtime session, the tokenizer, and the model download all mocked. `server/rerankerServiceHook.test.ts` covers startup error handling and shutdown cleanup with the service dependencies mocked.

`server/rerankerService.integration.test.ts` loads the real model and asserts ranking quality against English and Portuguese fixtures. It downloads ~136MB, the model plus a 17MB sentencepiece tokenizer, so it is excluded from the default suite:

```sh
npx vitest run --config vitest.integration.config.ts
```

## Error Handling

| Scenario | Behavior |
|----------|----------|
| Reranker not ready | Falls back to unranked SearXNG results |
| Model fails to load | Logged by the hook; reranker stays unready and search returns unranked results |
| Cached file of the wrong size | Logged, then downloaded again before the session is created |
| Download shorter than the reported size | Throws before anything is written, so the cache keeps no partial file |
| HuggingFace unreachable with a warm cache | Metadata failure is logged and the cached files are used unchanged |
| Empty documents array | Returns empty array without running inference |
| Unicode sanitization needed | Logs warning, continues with sanitized input |

## Observability

`/status` reports what reranking costs and what it changes, under `reranker`:
how long a rerank takes, the share of results that survive the score filter,
how often the percentage fallback has to rescue a batch, and how often a search
was served in SearXNG's own order because the model was not loaded or threw.
The existing top-level fields remain combined totals. `byType.text` and
`byType.images` each report `reranks`, `averageMs`, `considered`, `kept` and
`keptRate`, so text snippets and image titles can be assessed separately.
For example, two text reranks taking 100 ms and 300 ms and one image rerank
taking 20 ms produce:

```json
{
  "reranker": {
    "reranks": 3,
    "averageMs": 140,
    "considered": 22,
    "kept": 7,
    "keptRate": 31.8,
    "fallbackApplied": 1,
    "skippedUnhealthy": 0,
    "failed": 0,
    "byType": {
      "text": { "reranks": 2, "averageMs": 200, "considered": 20, "kept": 5, "keptRate": 25 },
      "images": { "reranks": 1, "averageMs": 20, "considered": 2, "kept": 2, "keptRate": 100 }
    }
  }
}
```

`averageMs` divides elapsed time by completed, non-empty reranks and rounds to
whole milliseconds; it includes score filtering and sorting. `keptRate` is the
percentage of considered results kept, rounded to one decimal place. Combined
values use summed durations and counts, not an average of the two category
averages or percentages. Empty categories report zeros, and an empty ranking
batch does not increment either category or the combined totals.

`fallbackApplied`, `skippedUnhealthy` and `failed` stay at the top level.
An unhealthy or throwing model increments the existing skip or failure counter
without counting a completed rerank. All counters reset on server restart.
The search endpoint passes an explicit `"text"` or `"images"` category through
ranking to the counters; it is independent of whether the top result is pinned.
Only the category, counts, duration and fallback flag are recorded, never a
query, URL or result text.

See the `/status` section of `docs/overview.md` for the full field reference.

## Related Topics

- **Search System**: `docs/overview.md` - Search pipeline overview
- **Server Hooks**: `docs/overview.md#server-hook-system` - Hook registration
- **Web Search Service**: `server/webSearchService.ts` - SearXNG integration