Right Tool, Right Job: What Cochrane’s AI Guidance Means For Evidence Synthesis Teams

Cochrane published an article in June 2026 addressing when AI tools should not be used in evidence synthesis. Right tool, right job: deciding when to not use an AI tool, authored by Ella Flemyng, Cochrane’s Head of Editorial Policy and Research Integrity, addresses a question distinct from most existing AI guidance: not how a tool should be used, but whether it should be used at all, with non-use treated as a valid and expected outcome.

The framework: responsible handover

The article builds on RAISE (Responsible use of AI in evidence SynthEsis), in particular RAISE 3 (version 3.1, March 2026), which covers selecting and using AI tools. It applies the responsible handover framework that RAISE 3 sets out. Before an AI tool is applied to a review, Cochrane specifies five questions that should be answered:

1. What is the tool’s purpose?
2. Where did the training, testing, and validation data come from?
3. Is the tool validated and performing well enough for the job?
4. Is it usable, and does the team have the capability to use it properly?
5. Is there transparency around licensing, availability, and documentation?

Working through these questions leads to one of three decisions. Reviewers can decide not to proceed at any point if they find red flags, proceed if the tool has been validated and is backed by strong evidence, or proceed with mitigations if the tool shows promise but its evidence is incomplete or its risks are moderate and can be monitored. Cochrane’s article lists several red flags, including no published validation in a context relevant to the review and terms of use that let the tool train on the reviewer’s content without an opt-out.

Cochrane’s article also states that tool performance is not the only relevant consideration. A tool that increases workload after implementation, without adequate support to resolve it, may be unsuitable.

Where AI is most helpful

Cochrane’s article points readers to RAISE 3 for a task-by-task view. RAISE 3 evaluates AI use across the stages of a review, including search, screening, extraction, and drafting, and rates the tools used for each task from “acceptable for use” to “not acceptable for use.” Tools rated “acceptable for use,” meaning their outputs can be used directly in the review as long as their limitations are acknowledged, include:

• The search. Rule-based tools that suggest search terms, chase citations, retrieve records, check a search against known studies, and remove duplicates, along with topic-modeling tools that map the main themes in a body of literature early in a review.
• Synthesis and visualization. Rule-based meta-analysis and evidence-mapping tools, whether standalone or part of a review platform, applied to data the review team has already extracted, with the reviewer setting the analytical choices.
• Certainty of evidence. Rule-based tools that guide reviewers through established frameworks such as GRADE, with the reviewers still making the certainty judgments.

Most of these tools follow fixed and often transparent rules, and they leave the judgment with the reviewer. The next section covers the tools where that is harder to guarantee.

Where AI is most dangerous

Cochrane’s article is specific about which tools, review stages, and warning signs carry the most risk.

Generative AI and LLM-based tools. Cochrane’s article states that, at present, every large language model or generative AI tool should “proceed with mitigations,” without exception. The reason is that these tools are what Cochrane calls “black boxes,” whose inner workings are not well understood. Under this classification, verification falls to the reviewer, either through human verification of every output or through a formal validation study conducted within the review, known as a study within a review (SWaR).

LLM synthesis and end-to-end tools. RAISE 3 rates only two uses of AI as “not acceptable for use.” These are generative LLMs that synthesize results across studies, and agentic tools that run an entire review from search to summary. In both cases, it judges the outputs too unreliable for any formal review.

Full-text screening and data extraction. CESAR, Cochrane’s platform study testing AI tools, has set specific stopping boundaries for screening and data extraction. For full-text screening, a tool can be dropped if its specificity falls below 50%. In data extraction, it can be dropped if more than 3% of the values extracted contain major errors. These thresholds matter because a missed exclusion or an incorrectly extracted value can impact a pooled effect estimate or a policy recommendation.

Tools with no independent validation, or no path to human oversight. Cochrane’s article treats these as warning signs for any tool, whatever its function. Its reasons not to proceed include performance claims based only on studies led by the tool’s developer, and inadequate human oversight, such as no way to monitor or audit the tool’s outputs.

Mapping Cochrane’s expectations to Nested Knowledge’s AI framework

In September 2024, prior to publication of RAISE 3, Nested Knowledge published NICE Guidelines for AI Methods: The Nested Knowledge Approach, describing three principles governing AI implementation on the platform: data provenance, methodological transparency, and human oversight. 

Cochrane’s article summarizes four expectations, set out in Cochrane’s position statement, for reviewers using AI in evidence synthesis: 

1. Reviewers remain responsible for their review, including the decision to use AI and how it is used.
2. Reviewers can use AI if they can show it will not compromise the rigor or integrity of the review.
3. Any use of AI is reported fully and transparently.
4. Human oversight is maintained whenever AI is used. 

These expectations correspond closely to Nested Knowledge’s principles. The correspondence is described below, including where it is partial.

Assessing tools before and during use. Cochrane’s article treats the first two expectations as the stage where reviewers assess an AI tool and show that it will not compromise the review, before deciding whether to use it. These correspond to Nested Knowledge’s documented data provenance and validated performance. Robot Screener, Nested Knowledge’s supervised screening classifier, has documented data provenance and internal validation performance across thousands of screened records. These figures are published on the AI validation and disclosure page.

RAISE 3 places emphasis on ongoing monitoring in addition to point-in-time validation. Robot Screener retrains on a team’s own screening decisions within each review, so its performance is cross-validated against reviewer decisions as the review progresses, rather than validated once and applied indefinitely. This gives reviewers the kind of review-specific evidence that RAISE 3 calls validation within the review. A formal SWaR goes a step further, with its own registered protocol, so that its results can be combined with those from other reviews.

Nested Knowledge’s LLM-based tools, including Smart Screener, Adaptive Smart Tags, and Smart Search, are classified under Cochrane’s current recommendation that all generative AI tools “proceed with mitigations.” Under this recommendation, the reviewer remains responsible for showing that the tool has not compromised the review, either through human verification of its outputs or through validation within the review. Nested Knowledge supports both routes. Reviewers can check every AI-generated output before it is used and report this in their disclosure of AI use. They can also run a human versus AI comparison for any LLM-based tool within a given project, which generates review-specific evidence of how the tool performed. This is distinct from CESAR. The comparison described here is generated by, and specific to, the individual review team using the tool.

Transparent reporting. Cochrane’s third expectation is transparent reporting of AI use, which corresponds to the audit trail in Nested Knowledge. Every AI action within a Nest, including tagging, screening, and extraction, is logged and traceable to source text and to the reviewer who confirmed it. This supports Cochrane’s disclosure requirement without requiring the audit trail to be reconstructed after the review is complete.

Human oversight. Cochrane’s fourth expectation is human oversight, which Cochrane’s article says follows from meeting the first three, because it rests on the reviewer’s deliberate and accountable decision to use AI. In Nested Knowledge, the workflow as a whole supports that oversight. Dual screening with adjudication, Study Inspector review of tagged and extracted data, and threshold-based application routed through a human decision are components of the standard review workflow rather than an added compliance step.

Why this standard applies to evidence synthesis

In practice, Cochrane’s article looks for three things in any AI tool used in a review. Its performance should have been properly checked, ideally including by someone other than its developer, its use should be reported in enough detail for others to reproduce the work, and reviewers must be able to oversee its output. These requirements follow from what a systematic review is used for. A review’s conclusions inform regulatory decisions, reimbursement determinations, and clinical practice guidelines. An error introduced during screening or extraction does not remain contained to that step. It can carry through into a pooled effect estimate, and from there into a decision made by a party with no visibility into how the underlying data were generated or checked.

A tool that gives one answer to one reader carries a different kind of risk. If that answer is wrong and nobody checks it, the damage stays in that one exchange. In a systematic review, an unchecked error reaches everyone who relies on that review. 

Nested Knowledge built its principles of data provenance, methodological transparency, and human oversight for this exact reason. Whatever tool a review team chooses, the questions are the same. Has its performance been checked, can its use be reported in full, and can a human catch its mistakes? A review is only as reliable as its least checked step, and that now includes any step an AI tool performs.