How Does Nested Knowledge Compare to Claude for Evidence Synthesis?

Introduction

The volume, complexity, and fast-moving nature of medical research complicates the synthesis of findings and literature. The conducting of systematic literature reviews (SLRs), network meta-analyses, and other evidence syntheses are time and labor intensive. Artificial Intelligence (AI) tools, particularly those that use Large-Language Models (LLMs), are looked to as a way to ease the burden of such analyses, as they provide an efficient way to pull in information, structure it, and generate high-level summarizations.

Modern AI tools embed LLMs within a system called a “harness” that gives the LLM context management capability and the ability to take actions (e.g. reading and writing files). These systems are commonly called agents and can be used in evidence synthesis for a variety of tasks. For instance, one can use an AI agent to create data extraction templates that the agent fills in or can use an agent to download a RIS file, convert it to a data table of titles and abstracts, and have the LLM read the table to make inclusions/exclusion decisions.

However, this flexibility can obscure the degree to which LLMs’ effectiveness differs based on the “harness” and workflow they are embedded in. Specialized tools like Nested Knowledge addresses limitations of LLMs by providing a harness tailored to the evidence synthesis workflow. In the case of Nested Knowledge, this includes full traceability for any AI extractions, in-review validation metrics, human-in-the-loop approval and audit capabilities, the ability to share living reviews that can be automatically updated and shared, copyright and usage rights compliance and improved tabular data extraction. Meanwhile, the LLMs used and LLM interactions within Nested Knowledge are built, monitored, and constantly improved with respect to SLR-specific workflows, unlike generalist agentic tools.

Nested Knowledge vs. Desktop Agents (e.g. Claude) 

Table A: Feature comparisons between Nested Knowledge and General AI Agents

Efficacy and Accuracy Improvements over General Agents

Improved Performance with Specialized Tools

One of the most important benefits to using specialized tools is that they sometimes can be shown to perform better on traditional accuracy and classification metrics. Although tools like Nested Knowledge may use the off-the-shelf OpenAI or Claude model as the LLM model in its workflow, its “harness” creates improved performance.

For example, forthcoming research from our team analyzed the performance of Claude (Max, Opus4.7) against Nested Knowledge (NK; v1.111, high fidelity mode) on accuracy in data extraction across five lung cancer trials for study and patient characteristics and efficacy outcomes. Although both tools exhibited high accuracy for study and patient characteristics, extraction of efficacy outcomes showed better performance for Nested Knowledge, with 79.6% of extractions correct under Nested Knowledge, compared to 68.3% for Claude. After accounting for missed subgroups by treatment arm, performance declined to 39.3% and 19.4%, respectively. Claude also showed greater variation across studies, with weaker performance in later-positioned outputs. 

Despite Nested Knowledge using an OpenAI LLM, and not a proprietary model, its harness allows the application to see performance improvements. For example, Nested Knowledge is optimized to produce a massive number of data point extractions, one at a time. This is in comparison to other agents, which may group up point extractions within a study or even across studies in arbitrary ways. Nested Knowledge also leverages cutting edge document extraction to machine readable formats (“High Fidelity Mode” in the software). Additionally, work has been done to optimize the harness for gathering and structuring the relevant/correct user input configuration and surfacing this high-relevancy information to the LLM to provide more precise instructions (e.g. column-level instructions, content types). 

Nested Knowledge’s Synthesis module, which includes various tools for summarizing and analyzing studies that have been screened and have had their data extracted, also can have benefits over general AI agents through careful use of the context window budget. Documents and texts provided to an LLM become part of that model’s “context window”. LLMs can pay more quality attention to data in their “context window”, resulting in improved accuracy on questions using specifically provided data versus relying on extraneous study data or its summarizations from context compaction. In the case of Nested Knowledge, Synthesis functions like Smart Insights and Synthesis Chat do not place the whole underlying study text into the LLM’s context-window, but instead extracts the relevant data into the LLM’s context window. This helps prevent the performance decline that can be seen by over-populating the context window.

Improved Traceability and Avoidance of Fabricated References

Specialized tools can help avoid fabricated references and lack of traceability of claims to underlying study text by keeping LLMs limited to insight generation based on included study data within a review. As noted above, having the tool focus on provided in-context data boosts performance. This includes utilizing techniques to limit an LLM’s summarization to those studies included in the literature review. This can include ensuring traceability, where any insight is tied to a specific quotation from the underlying study text. For example, when Nested Knowledge’s AI features returns a quote from the underlying source, the application uses deterministic matching for that actual text in the source, taking into account whether the LLM paraphrased or shortened the quote. An exact position of the quoted text is returned back to the user, thanks to every character and its exact position having been mapped by another element within the harness dedicated to this. Internal validation studies of the Nested Knowledge tool shows a 99% success rate linking the LLM’s rationale and citation to the underlying document. The image below outlines an example of how this works with the Smart Insights function, which allows an AI agent to develop analytic insights – with traceability to quotations in underlying studies.