What Is RAG? Retrieval-Augmented Generation for Lawyers

You already do RAG every time you Shepardize a citation by hand: you look up a source, then write something based on what it says. The acronym dresses up an old habit. A retriever fetches the passages most relevant to your question. A language model reads them. The model writes an answer that points back to the passages it just read.

That is what most lawyers really mean when they ask what is RAG legal AI. Retrieval-augmented generation, or RAG, is an AI architecture that grounds a language model in your own documents before it answers. The system searches a curated source set, hands the relevant passages to the model, and asks it to write only from what was retrieved. Citations point back to specific pages.

The recipe was named in 2020 by Patrick Lewis and a team of co-authors at Facebook AI Research, in a paper titled Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. They reported that RAG models “hallucinate less and generate factually correct text more often than BART,” the bare model they used as a baseline. Five years later, every legal AI vendor that matters runs some version of the same idea.

The marketing has gotten ahead of the architecture. Pablo Arredondo, who co-founded Casetext before Thomson Reuters bought it, put the practitioner version more plainly on a podcast: you “anchor the system in a search engine that will retrieve real results and then force GPT-4 to answer based on what it’s seeing in front of it, the real case law, not freestyling an answer.”

The distinction matters. A bare LLM is recall. It generates text from patterns it absorbed during training. RAG is research, then writing. The model still phrases the answer, but the substance comes from documents you control. For a litigator, that shift isn’t cosmetic. That’s RAG. The rest is engineering.

Word search is fast and dumb. It finds “loss of consortium” but misses “spousal companionship damages.” That’s what RAG is built to fix.

Retrieval is the part of RAG that decides which paragraphs the model reads before it writes. Two ideas matter here: vector embeddings and vector search.

A vector embedding is a list of numbers that acts as a numerical address for the meaning of a sentence. Sentences with similar meanings sit at addresses close together. Sentences with different meanings sit far apart. Pinecone’s framing is that “the semantic similarity of these objects and concepts can be quantified by how close they are to each other as points in vector spaces.” A vector database is the index holding those addresses, answering “which paragraphs are nearest to this query?” in milliseconds.

For a lawyer, meaning beats keywords. Asked for “communications suggesting price coordination,” a keyword search for “price-fixing” misses the email reading “let’s both hold the line at $42 next quarter.” A dense retriever finds it. A 2020 paper from Karpukhin and co-authors at Facebook AI showed dense retrieval beats classic keyword scoring by 9 to 19 percentage points in top-20 passage retrieval accuracy, because “synonyms or paraphrases that consist of completely different tokens may still be mapped to vectors close to each other.”

Pure dense retrieval isn’t enough for legal text on its own. Statutes, case captions, and Bates numbers are exact strings; embeddings smear them. Hybrid search is the fix: run a keyword scorer alongside dense retrieval and merge the ranked lists. Hybrid catches Smith v. Jones, 412 F. Supp. 3d 198, and the paraphrased holding three pages later.

Two engineering tricks make this fast and accurate. Specialized graph indexes scale similarity search to billions of passages at sub-second latency. Then a reranker rescores the top results with a slower model that reads the query and each passage together. Fast first, slow second, careful by the end. That two-stage shape is how an AI agent reads your case file in a serious tool.

RAG narrows the failure modes. It doesn’t close them.

The strongest evidence is Stanford’s 2024 audit by Varun Magesh, Daniel Ho, and co-authors at the RegLab and HAI. They tested Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI as those products existed in early 2024 and measured hallucination rates between 17 and 33 percent. Lexis+ AI was the most accurate at 65 percent of queries, Westlaw came in at 41 percent, and Practical Law at 19 percent. Vendors have shipped updates since, and no peer-reviewed re-audit has been published as of April 2026. Even so, the original numbers reset what “hallucination-free” can credibly mean.

The harder finding is what kind of error RAG produces. Misgrounded citations were the dominant failure mode: the cited authority is real and clickable, but the proposition the AI attributes to it isn’t actually in the case. The Stanford team wrote that those errors “are potentially more dangerous than fabricating a case outright, because they are subtler and more difficult to spot.” A fake case fails the most basic Shepardization. A misgrounded one fails only when you read the cite. That’s the kind of error that makes it into a brief.

By the time you’re reading vendor pitches, the question isn’t “does this use RAG” but “how does this use RAG, and how do you know it works.” Four questions cover most of the ground.

1. Show me a citation that links back to the source passage. Not a footnote. Not a list of “sources consulted.” A live link to the page where the cited proposition actually appears. If the demo cannot do this, the citation is decorative.
2. Tell me your evaluation set and your error rate. Magesh and her co-authors wrote that “until vendors provide hard evidence of reliability, claims of hallucination-free legal AI systems will remain, at best, ungrounded.” A serious tool has tested itself against a holdout set the vendor did not write. Ask for the methodology; insist on a number. Vendors that cannot describe their evaluation often cannot describe their data handling either, which raises the same privilege exposure in consumer AI that consumer chatbots already raised.
3. What happens when the answer isn’t in the corpus. The Vals AI Legal AI Report praised Vincent specifically for refusing to answer when the underlying database lacked the source. That refusal is a feature, not a bug. The wrong answer to “what should the tool do when it doesn’t know” is “make something up that sounds right.”
4. Who supervises the model when it’s wrong. ABA Op. 512 says “lawyers must have a reasonable understanding of the capabilities and limitations of the specific GAI technology that the lawyer might use.” A vendor that can’t describe how it surfaces error to your supervising attorney is a vendor that hasn’t finished building the product.