Retour
RAGCohere RerankBGE RerankerLLM EngineeringInformation Retrieval

I Tried Cohere and BGE Rerankers for Financial RAG. Here's Why Cohere Stayed in My Stack.

89% Utile11% Pas utile
0 Commentaires

The short version: In my work on a production financial-research assistant, Cohere Rerank 3.5 was my preferred relevance layer over the BGE reranking approaches I explored. That is a qualitative verdict from my use case, not a claim that Cohere tops every benchmark. BGE still has a compelling advantage when data control, self-hosting, or predictable infrastructure economics matters more than managed convenience.

A retrieval system can return documents that are semantically close to a question and still supply precisely the wrong evidence to an LLM.

In financial research, that difference is brutal. Ask for consolidated operating margin in Q2 FY25 and you may retrieve a perfectly similar paragraph about standalone margin, a different quarter, or management's general margin commentary.

The embedding has not necessarily failed. The document is related. It just does not answer the question.

That was the class of problem that made me take reranking seriously while working on Ask Iris, an AI-powered financial research assistant at Multibagg AI.

The broader system operated over 100,000+ documents and approximately 20 million records, and served 500,000+ user queries. It combined vector retrieval, metadata filtering, and LLM-based answering.

The distinction between plausible context and correctly targeted evidence mattered much more than another decimal place of cosine similarity.

I used Cohere reranking in the production retrieval stack, including cohere-rerank-3.5 through Pinecone's inference integration, and explored BGE-family rerankers as alternatives.

Of the approaches I tried, Cohere gave me the best practical relevance experience for that workload.

I did not preserve a controlled public head-to-head benchmark with nDCG, MRR, or latency figures, so I won't invent one here. Instead, this review explains what the models do mathematically, where the real tradeoffs sit, and what production incidents taught me.

1. Why the vector database's top result can be wrong

Most retrieval-augmented generation (RAG) systems start with a bi-encoder. The query and each passage are separately embedded into the same vector space:

q=fθ(q),di=fθ(di)\mathbf{q}=f_{\theta}(q),\qquad \mathbf{d}_i=f_{\theta}(d_i)

A common scoring function is cosine similarity:

sim⁡(q,di)=q⋅di∥q∥2∥di∥2\operatorname{sim}(q,d_i)=\frac{\mathbf{q}\cdot\mathbf{d}_i}{\lVert\mathbf{q}\rVert_2\lVert\mathbf{d}_i\rVert_2}

This separation is why dense search scales. Document vectors can be precomputed and indexed for approximate nearest-neighbor search. At query time, only the query needs a fresh embedding.

But compression has consequences.

The ranking must infer relevance from two fixed-size representations. Specific distinctions such as consolidated versus standalone, Q2 versus full-year, guidance versus reported result, or parent company versus subsidiary can get blurred.

The solution isn't to replace retrieval with an expensive model over the entire corpus. It's to let retrieval find a reasonably broad candidate set, then spend more computation judging the candidates against the exact question.

My preferred architecture is:

  1. Retrieve broadly with dense search, BM25, or a hybrid of both. Keep a candidate pool, say 40 passages, as a tunable starting point rather than a universal constant.
  2. Apply eligibility constraints before ranking when they are strict: company, report date, document type, or access permissions.
  3. Rerank each surviving candidate against the original user query.
  4. Feed only the strongest, sufficiently diverse and properly attributed evidence to the answering model.
  5. If evidence is missing or contradictory, say so. Do not manufacture certainty because a reranker produced a high score.

For hybrid search, I like reciprocal rank fusion (RRF) when lexical and vector scores live on incompatible scales:

RRF⁡(d)=∑m∈R1k+rank⁡m(d)\operatorname{RRF}(d)=\sum_{m\in\mathcal{R}}\frac{1}{k+\operatorname{rank}_m(d)}

Here, R\mathcal{R} is the set of retrieval systems, and kk is a smoothing constant, often chosen around 60, but it should be tuned.

RRF combines positions, not raw scores. It is not a substitute for reranking, just a useful way to assemble the shortlist.

The original RRF paper is still worth reading.

2. The mathematics: what a reranker actually learns

A typical encoder-style reranker is a cross-encoder.

Instead of embedding question and passage separately, it processes them jointly:

hi=Transformer⁡ϕ([CLS],q,[SEP],di,[SEP])\mathbf{h}_i=\operatorname{Transformer}_{\phi}([\mathrm{CLS}],q,[\mathrm{SEP}],d_i,[\mathrm{SEP}]) si=w⊤hi,CLS+bs_i=\mathbf{w}^{\top}\mathbf{h}_{i,\mathrm{CLS}}+b

Joint attention lets query words interact directly with document words.

A passage can contain all the words of a question and still receive a weak score if their relationships, qualifiers, or temporal context do not line up.

The BGE reranker family includes models built for this query-passage scoring pattern.

For example, the public BAAI/bge-reranker-v2-m3 can return raw relevance logits; the FlagEmbedding implementation can map these with a sigmoid:

pi=σ(si)=11+e−sip_i=\sigma(s_i)=\frac{1}{1+e^{-s_i}}

Important: Passing a logit through sigmoid does not automatically make it a calibrated probability. The number is useful for ordering documents, but its semantics need to be validated.

Ranking models can be trained with several objectives. These are general learning-to-rank formulations, not a disclosure of Cohere's proprietary training recipe.

Pointwise binary cross-entropy

Judge each query-passage pair separately.

Lpoint=−∑i[yilog⁡σ(si)+(1−yi)log⁡(1−σ(si))]\mathcal{L}_{\mathrm{point}}=-\sum_i\big[y_i\log\sigma(s_i)+(1-y_i)\log(1-\sigma(s_i))\big]

Pairwise logistic loss

Prefer a relevant passage d+d^+ to a less relevant passage d−d^-.

Lpair=log⁡ ⁣(1+e−(s+−s−))\mathcal{L}_{\mathrm{pair}}=\log\!\left(1+e^{-(s^+-s^-)}\right)

Listwise ranking

Optimize a distribution or ranking objective over a candidate list, rather than isolated pairs.

One illustrative softmax objective is:

Llist=−∑iPi∗log⁡esi/τ∑jesj/τ\mathcal{L}_{\mathrm{list}}=-\sum_i P_i^*\log\frac{e^{s_i/\tau}}{\sum_j e^{s_j/\tau}}

Here, Pi∗P_i^* is a target relevance distribution and τ\tau is a temperature.

Different implementations use different losses and sampling strategies.

This power has a price. A bi-encoder scores a query against indexed vectors cheaply. A cross-encoder generally needs a forward pass for each candidate pair, so the work grows approximately with candidate count multiplied by per-pair inference cost.

That is why reranking 40 candidates and reranking 40,000 candidates are entirely different system designs.

A third option worth understanding is late interaction, exemplified by ColBERT.

It precomputes contextualized document-token representations and scores the query with a token-level MaxSim operation:

S(q,d)=∑i∈qmax⁡j∈d⟨qi,dj⟩S(q,d)=\sum_{i\in q}\max_{j\in d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle

It occupies an interesting middle ground between cheap vector search and full query-document joint encoding, but brings its own indexing and serving complexity.

ColBERT paper.

3. What better ranking looks like, without fake benchmark numbers

Take this hand-constructed example, inspired by common financial-document ambiguity, not copied from a production test.

Query: Which filing reports consolidated operating margin for Q2 FY25?

CandidateSnippet typeHuman relevance grade
AEarnings commentary, wrong reporting period1
BStandalone Q2 note, wrong basis0
CConsolidated Q2 note, direct answer3
DPrior-year consolidated context1
EGeneric investor presentation0

Suppose dense search orders them A → B → C → D → E, and an illustrative reranker orders them C → A → B → D → E.

The same five documents reach the second stage, but the answer-bearing passage moves from third to first.

A useful metric here is normalized discounted cumulative gain (nDCG):

DCG@K=∑i=1K2ri−1log⁡2(i+1)\mathrm{DCG}@K=\sum_{i=1}^K\frac{2^{r_i}-1}{\log_2(i+1)} nDCG@K=DCG@KIDCG@K\mathrm{nDCG}@K=\frac{\mathrm{DCG}@K}{\mathrm{IDCG}@K}

Using the relevance grades above, the illustrative nDCG@5 rises from 0.606 to 0.991.

These are arithmetic results from a made-up example, not measurements of Cohere or BGE.

The calculation shows why ranking the right document first matters disproportionately.

For real model selection, I would report nDCG@5, MRR@10, Recall@N before reranking, and answer citation correctness.

MRR focuses on the rank of the first relevant item:

MRR@K=1∣Q∣∑q∈Q1[rq≤K]rq\mathrm{MRR}@K=\frac{1}{|Q|}\sum_{q\in Q}\frac{\mathbb{1}[r_q\le K]}{r_q}

Here, rqr_q is the first relevant result's rank; the term is zero when no relevant item appears within KK.

One crucial guardrail: reranking cannot recover an answer that was absent from the candidate pool.

If recall@40 is poor, improving the second stage alone won't fix it.

4. Cohere versus BGE: my qualitative verdict

The answer is not simply “proprietary beats open-source.”

I found Cohere more convincing for the financial queries I cared about, but the deployment tradeoffs are real.

DimensionCohere Rerank 3.5BGE family, e.g. bge-reranker-v2-m3
Relevance in my workloadMy preferred choice, qualitativelyUseful alternative; no published head-to-head score from my testing
IntegrationManaged API or supported platform integrationRun locally or on your own inference infrastructure
Model transparencyProprietary weights and training detailsPublic model weights and configurable inference
OperationsLess model-serving work, but provider dependencyMore control, but you own capacity, batching, upgrades, and uptime
CostsUsage-related API/platform chargesCompute, memory, engineering time, and infrastructure utilization
Best fitTeams optimizing quickly for relevance and shipping speedTeams prioritizing control, data locality, custom tuning, or sustained high utilization

BGE is not one fixed model.

The FlagEmbedding repository describes bge-reranker-base, large, v2-m3, and newer variants.

Compare specific checkpoints on identical candidates. A model-family label is not a reproducible experiment.

Why did I prefer Cohere?

My practical evaluation was tied to whether the selected evidence matched the actual intent of a financial query, not merely whether the document looked related.

A reranker that distinguishes the correct fiscal period, reporting basis, and metric is more valuable than one that returns another polished but misleading excerpt.

Cohere was the one I favored for that application.

Why might I still choose BGE tomorrow?

Privacy restrictions, very high steady-state query volume, a strong GPU-serving stack, domain-specific fine-tuning, or a requirement to avoid external network calls can flip the decision.

The best reranker is an accuracy-latency-cost-reliability decision, not a leaderboard trophy.

5. The production failure that mattered more than a leaderboard

A difficult lesson came from the integration, not necessarily the neural model.

In a previously working Pinecone inference setup, our cohere-rerank-3.5 requests began returning a 403 / PERMISSION_DENIED because the project was no longer authorized to access that model.

Retrieval still needed to function even when the reranking dependency did not.

There was also a separate reranking issue involving an empty-document input and environment configuration.

Neither failure tells us that Cohere's neural relevance model was bad. Both tell us that a relevance layer is still a production service with permissions, inputs, quotas, and failure modes.

My defensive checklist now looks like this:

  • Validate inputs: Drop empty text, preserve document IDs, and record why candidates were filtered.
  • Keep authorization and model access observable: A 403 should be classified differently from a timeout or malformed input.
  • Use a bounded timeout and fallback: Return the original retrieval ordering if the reranker is unavailable, and mark the response internally as degraded.
  • Preserve provenance: Attach company, fiscal period, source document, page, and passage IDs before calling the reranker. Restore the metadata after ranking using the returned indices.
  • Evaluate degradation: A functional fallback is not a relevance-equivalent fallback. Measure its answer/citation impact.

Another subtle failure is treating rerank scores as universal probabilities.

Cohere's documentation cautions that ranking scores are query-dependent, and older model scores should not be casually compared across unrelated queries.

The newly released Rerank 5 family explicitly advertises calibrated scores; don't backport that guarantee to v3.5.

Cohere best practices.

6. Reproduce the comparison on your own documents

A minimal side-by-side test can be written in Python.

It compares the same candidate passages, with no retrieval differences to contaminate the result.

# pip install cohere FlagEmbedding
import os
import cohere
from FlagEmbedding import FlagReranker

query = "Which filing reports consolidated operating margin for Q2 FY25?"
docs = [
    "Management discussed margins across the full fiscal year.",
    "Standalone operating margin for the second quarter is reported here.",
    "Consolidated operating margin for Q2 FY25 is disclosed here.",
    "The prior-year consolidated statement provides comparison context.",
]

# Cohere v3.5 was the generation I used in production.
client = cohere.ClientV2(api_key=os.environ["COHERE_API_KEY"])
co = client.rerank(model="rerank-v3.5", query=query,
                   documents=docs, top_n=len(docs))
cohere_order = [item.index for item in co.results]

# Public checkpoint chosen as a reproducible BGE-family baseline.
bge = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=False)
bge_scores = bge.compute_score([[query, text] for text in docs])
bge_order = sorted(range(len(docs)),
                   key=lambda i: float(bge_scores[i]), reverse=True)

print("Cohere order:", cohere_order)
print("BGE order:   ", bge_order)

This is a runnable comparison recipe, not a benchmark result.

It needs a Cohere API key, local model dependencies and hardware sufficient for BGE, and a real test set.

To turn it into publishable evidence, freeze your corpus, candidate pools, model versions, truncation policy, and relevance labels.

Run at least dozens of genuinely representative queries, including hard negatives with the wrong dates, entities, or reporting basis.

Then report per-query wins and failures rather than only a pooled average.

My suggested experiment matrix would keep the first-stage top 40 fixed and compare three rankings: raw vector/hybrid, Cohere, and BGE.

Measure nDCG@5, MRR@10, p50/p95 rerank latency, request failures, billed units or GPU-seconds, and whether the final answer cites the correct passage.

The exact 40 and 5 are starting points to tune, not supposedly optimal numbers.

For papers with stronger public evaluation discipline, start with BEIR. It demonstrates why out-of-domain evaluation is necessary and why stronger ranking can come with higher computational cost.

7. The cost math most reranker comparisons omit

“BGE is free” is as misleading as “Cohere is expensive.”

BGE model weights may be freely available under their applicable license, but inference hardware, utilization, operations, and engineering time aren't free.

Cohere pricing also requires understanding the search unit, not just the model name.

Cohere defines a search unit as one query with up to 100 billable documents. Longer documents can be split into separately counted chunks; the exact billing and platform terms must be checked for your provider.

Cohere pricing FAQ.

For illustration, a third-party listing quotes $2 per 1,000 searches for Rerank 3.5 in October 2026.

That is a point-in-time reference, not a guaranteed contractual rate.

Vercel model listing.

If each request is one billable search unit, the arithmetic is:

CAPI=Qmonth×Uper query1000×P1000C_{\mathrm{API}}=\frac{Q_{\mathrm{month}}\times U_{\mathrm{per\ query}}}{1000}\times P_{1000}

With 10,000 user searches/day, 30 days, one billable unit/search, and $2/1,000 units:

CAPI=10,000×30×11000×$2=$600/monthC_{\mathrm{API}}=\frac{10{,}000\times30\times1}{1000}\times\$2=\boxed{\$600/\mathrm{month}}

That $600 is illustrative, not my actual production invoice.

Requests that consume more than one billed unit, provider-specific pricing, or additional search/retrieval costs change the result.

A self-hosted option needs a different calculation:

Cself=Caccelerator+Cserving+Cops+CreliabilityC_{\mathrm{self}}=C_{\mathrm{accelerator}}+C_{\mathrm{serving}}+C_{\mathrm{ops}}+C_{\mathrm{reliability}}

The break-even point is when the measured fully loaded self-hosted cost per successfully reranked query falls below the hosted alternative at the same relevance and latency target.

A GPU that sits idle most of the day can be a worse deal than an API; well-batched, fully utilized serving may change that equation.

8. The new contender I have not tested: Cohere Rerank 5

On October 21, 2026, Cohere released Rerank 5 Pro and Rerank 5 Fast, advertising a 32k-token context window, support for 100+ languages, calibrated scores, and Context Focus that returns relevant character spans inside documents.

Cohere positions Pro for complex retrieval and Fast for throughput-sensitive workloads.

These are vendor-reported product capabilities, not findings from my comparison.

Official release notes.

For a financial RAG system, Context Focus is especially interesting: it promises a way to pass only answer-bearing excerpts downstream instead of dumping full filings into a generator.

But I would validate whether the selected spans preserve footnotes, table headings, units, and page-level provenance before relying on them.

Saving tokens at the cost of losing accounting context is a bad optimization.

I would rerun the same labeled-query evaluation before replacing v3.5 or a self-hosted baseline with v5.

No upgrade is a free quality guarantee.

My final verdict

What impressed me: Cohere Rerank 3.5 was the best practical relevance fit among the reranking approaches I explored for my financial-document assistant. It made it easier to focus generation on the right evidence, not merely neighboring text, without owning another inference service.

What fell short: Managed convenience comes with provider dependency. I experienced a permissions failure and separate input/configuration issues that made fallback design and observability non-negotiable. BGE's self-hosting story remains genuinely attractive.

Worth it? Yes, for a team that values relevance and iteration speed and can afford a managed API. Also yes to BGE when deployment control or measured cost economics are the priority.

The better technical decision is the one that wins on your labeled domain queries, meets p95 latency targets, and fails safely in production.

The larger lesson: A good RAG system is not the one that retrieves the most text. It's the one that puts the right evidence in the right order, then knows when the evidence isn't there.


References and reproducibility notes

Disclosure: This is an independent engineering review based on my experience and publicly linked technical documentation. The numerical nDCG example and monthly cost scenario are explicitly illustrative. No proprietary evaluation results or private customer documents are reproduced.

Commentaires0

Connectez-vous pour commenter
  • Pas encore de commentaires.