Overview
Source Trust Scoring assigns dynamic trust scores to every document and source based on consistency, citation track record, freshness, and contradiction rate. High-trust sources rank higher in retrieval; low-trust citations trigger warnings.
This addresses a fundamental flaw in RAG systems: treating all sources as equally reliable, regardless of their provenance, age, or accuracy history.
The Problem
- 47% of enterprise users have made major decisions based on hallucinated or unreliable content — partly because RAG treats all sources equally
- Document corpora inevitably contain a mix of authoritative policy docs, informal meeting notes, outdated drafts, and potentially inaccurate materials
- No mainstream RAG tool differentiates between a recently updated official handbook and 14-month-old meeting notes
- Users can't see why the system retrieved a particular source or assess its reliability
- Over time, contradictions accumulate between documents without any mechanism to surface or resolve them
How It Works
- Freshness score — Based on document age and update frequency for the topic area
- Consistency score — How often this document's claims align with or contradict other documents on the same topic
- Citation score — How frequently this document is cited by inferences that are later validated by users
- Authorship score — If metadata includes author/source provenance, factor in historical reliability of that source
- User feedback — Users can flag documents as authoritative or unreliable; these overrides feed into scoring
- Retrieval integration — Trust score becomes a weighted factor in hybrid retrieval ranking
- Decay — Scores decrease for documents in fast-moving topic areas that haven't been updated recently
User Story
Your corpus has 200 documents. The system scores each: "Company Handbook v4.2 — Trust: 94/100 (consistent with other sources, frequently cited, recently updated). Meeting Notes Jan 2025 — Trust: 42/100 (contradicts 2 other documents, never cited by validated inferences, 14 months old)." When a user asks a question in Chat, the system preferentially retrieves from high-trust sources. When an inference cites the meeting notes, a subtle warning appears: "This claim draws from a lower-trust source — consider verifying." Over time, as users validate or dispute inferences, the trust graph becomes more accurate.
Complexity & Timeline
| Aspect | Detail |
|---|---|
| Complexity | Medium |
| Estimated Build | 3–4 weeks |
| Platform Dependencies | Metadata extraction, Retrieval, Inferences (citation tracking) |
| New Infrastructure | Trust scoring engine, contradiction detection, user feedback schema, retrieval weight integration |
Target Clients
- Personas: Chief Knowledge Officers, Data Governance Leads, Risk Managers, Research Directors
- Verticals: Financial Services, Legal, Healthcare, Government, any organization with document quality concerns
- Pitch: "Not all sources are equal — know which ones to trust and which ones to verify."
Revenue Potential
Strong differentiation: no mainstream RAG tool has source-level trust scoring. Foundation feature that enhances every other retrieval-dependent capability — its value compounds across the platform. Particularly attractive to regulated industries where source reliability has legal implications. Supports premium "governance tier" positioning. Creates a virtuous cycle: as users provide feedback, trust scoring improves, making the system more valuable over time.
Feature Synergies
- Decision Journal — Decision outcomes feed back into source trust scores, creating a closed learning loop
- Compliance Mapper — Weight compliance evidence by source reliability; flag requirements covered only by low-trust documents
- Expert Fingerprinting — Documents authored by recognized domain experts receive higher initial trust scores
Risks & Open Questions
- Trust scoring introduces a "rich get richer" dynamic — highly cited documents gain trust, potentially suppressing newer, better sources
- Contradiction detection between documents is computationally expensive and error-prone for nuanced topics
- User feedback can be gamed or biased — need safeguards against individuals artificially inflating or deflating scores
- Transparency challenge: users need to understand why a source has a particular trust score without the UI becoming cluttered