ENGINEERING CASE STUDY / 2026
FinRAG
Evidence-first financial document question answering, built as a local research system rather than a hosted chatbot.
FinRAG keeps the evidence visible: it ingests PDFs, retrieves with TF-IDF, MiniLM and Multi-Query, checks reporting periods, generates answers locally with Qwen, and returns the passages and page references behind each response. The repository records retrieval coverage, tests and the limits of those measurements.
Published on main. Local research application. Not a production financial-advice tool.
OVERVIEW
What was built, how it was validated, and what remains limited
FinRAG is a local financial-document question-answering system. Python owns extraction, retrieval, evidence handling, period checks and generation; React displays the result and keeps evidence attached to the original response. Recorded holdout retrieval coverage is 20.6% for MiniLM and 26.5% for Multi-Query. Those are retrieval metrics, not answer-accuracy measurements.
- Built: PDF ingestion, TF-IDF/MiniLM/Multi-Query retrieval, evidence fusion, local Qwen generation, FastAPI service and React workspace.
- Validated: 146 Python tests, Ruff, 9 web tests, 17 browser tests, production frontend build and FinanceBench retrieval experiments.
- Limited: citations are inspectable rather than fact-verified; Qwen may abstain or reason incorrectly; retrieval coverage on holdout remains low.
KEY ENGINEERING DECISION
Local inference over API dependence
FinRAG uses a local Ollama model instead of a paid API. This supports privacy, reproducibility and offline inspection, but generation quality, latency and capability are constrained by the local model. The implementation keeps retrieval evidence available even when generation fails.
PROBLEM AND DECISIONS
Retrieval is not the same as a correct answer.
A financial answer may require facts from several pages, the correct reporting period and arithmetic. Finding passages that look relevant is not evidence that the generated answer is right.
Problem
The system must answer questions about financial reports while keeping the evidence visible. It cannot treat a plausible model response as proof. It also has to distinguish missing evidence, abstention, malformed output and backend failure instead of collapsing them into one error message.
Constraints
- No paid LLM API or API key for inference.
- Local, single-user research application; not an internet-facing service.
- PDF extraction depends on selectable text; no validated OCR pipeline.
- FinanceBench data is CC BY-NC 4.0; reports and annotations are not redistributed.
- Models are pretrained and are not fine-tuned by this project.
ARCHITECTURE
Python owns the evidence; React displays it.
The backend performs extraction, retrieval, evidence handling, period checks and generation. The frontend does not recompute retrieval or financial figures.
INGESTION
25/5 direct · 80/15 page
RETRIEVAL AND GROUNDING
API retrievers
plan, rank pages, RRF, neighbours
reporting context only
DELIVERY
status, evidence, timings
evidence, PDF pages, states
Implementation components
- Ingestion:
pypdfpage extraction; word-based chunks. - Retrieval:
TfidfVectorizercosine similarity and Sentence Transformers MiniLM. - Evidence: chunk formatting, distinct-chunk selection and page-neighbour selection.
- Generation: local Ollama
qwen3:4b-instructat temperature 0. - API: FastAPI, Pydantic strict request models, upload validation, sandboxed PDF responses.
- Stack: Python 3.13, React, Vite, pytest, Ruff, Playwright.
RETRIEVAL AND EVIDENCE
Multi-Query is page fusion, not a prompt trick.
Direct retrieval returns top chunks. Multi-Query expands a financial question into evidence-oriented searches, ranks pages from each search, fuses the page rankings and then selects neighbouring chunks in reading order.
Plan queries
The original question is retained. Recognised financial concepts add statement-heading searches.
Search all chunks
Each query searches the fitted retriever over all chunks in the selected document.
Rank pages
Chunks are grouped by physical page. Each page keeps its highest chunk score.
Fuse rankings
With more than one query, page rankings are combined with Reciprocal Rank Fusion using constant 60.
Select neighbours
For each top page, the code selects the nearest chunk IDs around the best-scoring chunk.
Return evidence
The result is a page-grounded evidence set with chunk text, scores and page indexes.
Chunking settings
| Mode | Chunk size | Overlap | Unit |
|---|---|---|---|
| Direct | 25 | 5 | words per physical page |
| Multi-Query | 80 | 15 | words per physical page |
These are API ingestion settings. They do not change the historical evaluation scripts or algorithms.
Query planning example
For a quick-ratio question, the code adds searches for balance-sheet cash, short-term investments, accounts receivable and total current liabilities.
FINANCIAL_CONCEPTS = {
"quick ratio": [
"consolidated balance sheet cash short-term investments accounts receivable",
"consolidated balance sheet total current liabilities",
],
}Response states
produced | Passed structural checks and citation range checks. |
model_abstention | Model emitted an abstention marker. |
invalid_output | Missing, malformed or out-of-range citations. |
no_evidence | No evidence supplied. |
insufficient_context | Requested period not verified in recognized reporting context. |
backend_error | Ollama or network failure; evidence is still returned. |
Period grounding
The period checker searches complete evidence-document pages for explicit fiscal periods or supported English statement dates. It can add recognized reporting-context headers to model evidence.
- Requested years found in recognized context:
available. - Requested years missing from recognized context:
insufficient_context. - Unknown date format:
unverified, query is not blocked. - It does not verify amount-to-column alignment, units or arithmetic.
EVALUATION AND RELIABILITY
Recorded retrieval coverage, not answer accuracy.
The metrics below are retrieval evaluations. They measure whether annotated supporting pages are recovered, not whether the generator uses them correctly.
FinanceBench all-evidence Hit@5
| Split | Questions | Reports | MiniLM | Multi-Query |
|---|---|---|---|---|
| Pilot | 28 | 6 | 46.4% | 53.6% |
| Holdout | 34 | 10 | 20.6% | 26.5% |
All-evidence Hit@5 requires every annotated supporting page to appear among the five highest-ranked distinct pages for the relevant report. It is stricter than finding at least one supporting page and does not assess answer correctness.
Synthetic regression corpus
| Retriever | Accepted chunk @1 | Accepted chunk @3 |
|---|---|---|
| TF-IDF | 83.3% | 100.0% |
| MiniLM | 75.0% | 100.0% |
4 documents, 12 chunks and 12 answerable questions, plus 6 unanswerable questions. This is an any-evidence hit rate used to verify retrieval code, not a FinanceBench result.
Failure analysis
In 39 real Ollama inferences across several evidence and prompt conditions, retrieval supplied the required figures for a financial calculation, while generation could abstain, omit components or describe arithmetic incorrectly. This motivates the strict separation between retrieving evidence and verifying numeric claims.
Recorded checks and boundaries
PYTHONPATH=src python -m pytest -q | 146 passed, one third-party deprecation warning |
ruff check . | Passed |
cd web && npm test | 9 passed |
cd web && npx playwright test | 17 passed, mostly mocked HTTP |
cd web && npm run build | Passed |
Playwright tests are not a browser-to-real-Ollama accuracy benchmark. Backend generation errors return HTTP 200 with evidence and a backend-error state; malformed requests return 422 and unavailable services return 503.
LIMITATIONS AND SOURCE
The boundaries are published with the results.
These limits are taken from the repository documentation, not added as an afterthought.
- Citations are inspectable, not automatically fact-verified.
- Qwen may abstain despite sufficient retrieved evidence; a non-abstaining answer may still be wrong.
- Financial numbers and arithmetic should be checked against the original report.
- Retrieval scores are ranking signals, not probabilities of factual correctness.
- PDF extraction can lose table structure and no OCR pipeline is validated.
- The app is intended for local, single-user research; it has no hardened authentication or deployment security model.
- Pilot and holdout results come from limited question/report subsets and do not establish general financial-answer accuracy or production readiness.
LINKS
Source and related work
CONTACT
Have a hard problem?
Let's make it measurable.
Open to AI Engineer, Applied ML and Software Engineer roles where evaluation and engineering judgment matter as much as model choice. Based in Madrid.
Open to AI Engineer, Applied ML and Software Engineer roles