FinRAG
Multimodal Retrieval for Financial Documents
4
Modalities, One Vector Space
text · PDF · image · audio
3072 / 768
Named Vectors per Point
quality or cost, per query
SHAP
Result Explanations
token-level attribution
01The Challenge
FinRAG started from a question that comes up in every financial NLP engagement: why do retrieval systems struggle with financial documents?
Part of the answer is structure. Filings, transcripts and proxy statements are among the most information-dense documents in existence — dense prose, nested tables, footnotes that override headline numbers, and cross-references spanning hundreds of pages.
But the harder part is that the evidence is not all text. A number lives in a table, the qualification that changes its meaning is said aloud on the earnings call, and the trend is only legible in a chart. The usual answer is to caption the image, transcribe the audio, and embed the resulting text — which means every cross-modal query is answered against a lossy paraphrase rather than the thing itself.
FinRAG takes the other route: embed each modality directly into one shared vector space, so a text query can reach a chart or an audio segment without passing through a description of it. It is live at finrag.io.
What this system is, precisely. FinRAG is the retrieval layer of a RAG stack, not the whole stack. It finds and ranks evidence and shows why it ranked it; it does not currently generate written answers on top. That is a deliberate order of work rather than an omission — retrieval is the half that decides whether a RAG system can be right at all, and a generation layer built over weak retrieval only makes its failures more fluent.
02Our Approach
Ingest four modalities, chunk each by its own natural unit, embed them all with one model into a single vector space, index with two resolutions per point so cost can be traded against quality per query, search with the result budget split across modalities so one loud modality cannot crowd out the rest, and attribute each result back to the tokens that drove it.
- Google Gemini Embedding 2 — one model that embeds text, PDF pages, chart images and audio into a single shared space — no captioning or transcription step in between, which is the usual place multimodal pipelines lose fidelity
- Qdrant — vector store holding two named vectors per point — a 3072-dimension full vector and a 768-dimension compact one — with payload filtering by modality
- PostgreSQL — document and chunk metadata via async SQLAlchemy and Alembic migrations, kept separate from the vector store so the corpus can be re-indexed without losing provenance
- Cloudflare R2 — object storage for source documents and extracted assets; nothing is served directly, only through time-limited presigned URLs
- SHAP — optional explainability service producing token-level attributions for a result — loaded behind a feature flag because it carries a transformer model and roughly a gigabyte of memory
- FastAPI + Next.js — async Python API on Railway, React frontend on Vercel at finrag.io
Ingest
PDF, text, image, audio
Chunk
512-word window, 128 overlap
Embed
one shared vector space
Index
Qdrant, dual named vectors
Search
per-modality balanced
Explain
SHAP attributions
03Key Findings
One embedding space, no captioning bridge
Text, PDF pages, chart images and audio are embedded by the same model into the same space. The alternative — caption the image, transcribe the audio, embed the text — introduces a paraphrase between the query and the evidence, and every cross-modal match is then only as good as that paraphrase. Removing the bridge is the central architectural bet of the system.
Two vector resolutions per point
Each point carries both a 3072-dimension vector and a 768-dimension one, stored as named vectors in the same collection. A query picks its resolution: the compact vector for cheap, high-volume search, the full vector when quality matters more than cost. The same lever would otherwise require a second index.
Per-modality balanced search
The result budget is divided across the requested modalities rather than allocated by raw score. Without this, whichever modality happens to produce higher similarity scores dominates every result set, and a multimodal index quietly degrades into a single-modality one.
Explanations as a first-class surface
A retrieval score is a number with no argument attached. The explain service returns token-level SHAP attributions for a result, so a user can see which parts of a passage drove the match. It sits behind a feature flag because it loads a transformer and roughly a gigabyte of memory — worth paying when a human is auditing a result, not on every query.
What has not been measured yet
This case study deliberately reports no retrieval precision, recall or latency figures. The system has unit and integration tests, but no standing retrieval evaluation — no labelled query set, no held-out relevance judgements. Publishing benchmark-shaped numbers without that harness behind them would be worse than publishing none, so a labelled evaluation set and a reproducible retrieval benchmark are the next piece of work, and any figures will appear here only once they are measured.
Interactive Demo
Try FinRAG Live
Search a demonstration corpus of financial text, filings, charts and earnings audio — filter by modality and inspect why each result matched.
Launch FinRAG04Business Impact
Projected Annual Value
A multimodal retrieval layer that searches text, filings, charts and audio together — and shows its reasoning
The value of FinRAG is what it makes reachable. A question about a figure can surface the filing page it sits on, the chart that plots it, and the moment on the earnings call where it was qualified — from one query, without anyone having decided in advance to transcribe the call or caption the chart in a way that happened to preserve the relevant detail.
It is built as a portfolio system and is deployable against any document corpus. It currently indexes a demonstration set spanning the four supported modalities. FinRAG pairs with market-sentiment.io to form DataSalt's financial AI demonstration suite, and the mathematics underneath it — retrieval geometry, ranking, fusion and evaluation — is written up in depth at formalrag.com.
05Technical Details
Ingestion & Chunking
- PDF handling via pypdf; documents split into page ranges for embedding
- Text: 512-word sliding window with 128-word overlap
- Audio: fixed-length segments with overlap, embedded directly rather than transcribed
- Images: embedded whole, as a single chunk
- Source files and extracted assets stored in Cloudflare R2 (MinIO locally)
Embeddings — the multimodal core
- Model:
gemini-embedding-2-previewvia the google-genai SDK - Modalities: text, PDF, image and audio in one shared space
- Dimensions: 3072 full, 768 compact — both produced per chunk
- Calls wrapped in a thread pool with exponential backoff for rate limits
Retrieval
- Vector store: Qdrant, two named vectors per point, selected per request
- Filtering: Qdrant payload conditions on modality and document
- Balancing: the result limit is divided across the requested modalities
- Results carry a time-limited presigned URL rather than a direct object link
- No sparse leg, no fusion and no reranking stage — dense retrieval only
Explainability
- Token-level SHAP attributions over a result, surfaced in the UI as a waterfall, a bar chart and inline highlighting
- Ships as an optional dependency group and is gated by an environment flag; the import degrades gracefully when absent
Infrastructure
- Backend: FastAPI on Railway, with Alembic migrations run before each deploy and a health-check endpoint
- Frontend: Next.js App Router on Vercel at finrag.io
- Metadata: PostgreSQL via async SQLAlchemy and asyncpg
- Vector store: Qdrant; object storage: Cloudflare R2
- Demonstration audio was generated with Gemini TTS as an ingestion input — the system embeds audio, it does not read answers aloud
Not built (yet)
- Answer generation, citation rendering and response streaming — the retrieval layer is complete, the generation layer above it is not
- Sparse retrieval and rank fusion — a single dense leg today
- Cross-encoder reranking over the candidate set
- A labelled retrieval evaluation set and a standing benchmark
Facing similar challenges?
Let's discuss how data science can drive results for your business.