> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stellarsight.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Search evaluation

> nDCG@10 0.864 over a 50-query graded set, with method, caveats and a CI regression gate.

Generated by `npm run eval:search`. Reproduce from a clean clone with
`npm install && npm run eval:search`.

| Metric           | Value      |
| ---------------- | ---------- |
| nDCG\@10         | **0.864**  |
| Recall\@20       | **0.9045** |
| MRR\@10          | **0.9201** |
| Precision\@1     | 0.8958     |
| No-match silence | 0.5        |

Corpus: 27 records from `packages/index/src/seed.mjs`, **known-item**.
Queries: 50 (48 judged, 2 no-match probes).

## What these numbers are not

The corpus is the seeded demo catalog and the judgments were written by the same person
who wrote the ranker. Both facts inflate confidence and neither is hidden: this is a
known-item measurement, the honest floor for "does the ranker work at all", and the
Tranche 1 deliverable replaces it with 150–200 queries plus a rolling sample drawn from
the live catalog, which nobody here authored.

`no-match silence` is the fraction of deliberately unanswerable queries
(`quantum teleportation as a service`) for which the ranker returns **nothing** rather
than its best bad guess. A low number here is a real weakness, published on purpose:
BM25 will happily match a stray token, and the cold-start section of
[SEARCH-QUALITY.md](https://github.com/pedro-pelicioni/stellarsight/blob/main/docs/SEARCH-QUALITY.md) names the mitigations that are not built yet.

## Weakest queries

| Query                            | Intent     | nDCG\@10 | Top hit                                                   |
| -------------------------------- | ---------- | -------- | --------------------------------------------------------- |
| `will it rain tomorrow`          | paraphrase | 0        | —                                                         |
| `logistics cost estimation`      | paraphrase | 0        | —                                                         |
| `verify a tax id`                | task       | 0.3374   | `https://api.registry.example/v1/company/12345678000190`  |
| `detect fraudulent transactions` | paraphrase | 0.5      | `https://mcp.stellartools.example/mcp#submit_transaction` |
| `read soroban contract events`   | task       | 0.5      | `https://mcp.stellartools.example/mcp#simulate_contract`  |

## Method

* `gain(rel) = 2^rel - 1`, grades 0–3, judged in `eval/golden.jsonl`.
* nDCG\@10 against the ideal ordering of the judged set; queries with no judged document
  are excluded (nDCG is undefined for them) and counted under no-match instead.
* Recall\@20 and MRR\@10 count a document as an answer at grade **>= 2**. Grade 1
  ("marginally related") therefore contributes to nDCG and not to the binary metrics.
* Every query runs through `catalog.search`, the same call `/discovery/search` serves.

## Regression gate

`npm run eval:search` compares against `eval/baseline.json` and exits non-zero if any
metric falls more than 0.02 below it. CI runs it on every push, so a ranking change
that quietly costs relevance fails the build instead of shipping.