Model Name: Smart Critical Appraisal (SCA) #
Version: 1.0 #
Overview #
Smart Critical Appraisal (SCA) is a large-language-model–assisted appraisal system designed to support structured quality assessment of included studies within systematic review workflows.
Integrated directly into Nested Knowledge’s AutoLit platform, SCA helps reviewers evaluate methodological quality, risk of bias, and study reliability using established appraisal frameworks. The system reviews full-text study PDFs against checklist-based critical appraisal questions and generates recommended responses with supporting annotations drawn from the source text.
Rather than replacing expert review, SCA is designed to reduce the manual burden of appraisal while preserving reviewer control. AI-generated answers are presented within the structured appraisal workflow, can be reviewed and edited by humans, and never overwrite existing human decisions.
SCA is currently compatible with selected checklist systems and operates only on included studies with uploaded full texts, ensuring that appraisal recommendations are grounded in the full study record.
The system is intended for evidence synthesis teams seeking to accelerate critical appraisal while maintaining methodological transparency, auditability, and human oversight.
Intended Use #
Primary Purpose #
To assist reviewers in applying structured critical appraisal frameworks to full-text studies by generating evidence-linked recommended responses to checklist questions.
Intended Users #
- Systematic review teams
- Clinical researchers
- Medical affairs teams
- HEOR teams
- Regulatory and market access teams
- Evidence synthesis organizations conducting structured literature reviews
Limitations #
- Outputs depend on the completeness and clarity of reporting in the full-text article.
- Some appraisal questions require interpretive methodological judgment that may not be fully recoverable from text alone.
- Performance varies by appraisal framework and study type.
- SCA is intended to support, not replace, expert appraisal.
Training Data #
- Dataset:
SCA relies on a general-purpose large language model provided by OpenAI. The underlying model is trained on a mixture of licensed data, data created by human trainers, and publicly available text. - Validation Dataset:
Internal benchmarking datasets based on expert-reviewed studies appraised using supported critical appraisal systems. - Inference Inputs:
For each study, SCA evaluates: - The full-text PDF
- Extracted article text
- The selected appraisal framework and checklist questions
- Optional configured appraisal scope, where applicable
- Language:
Primarily English.
SCA does not perform fine-tuning on customer data. User documents are processed transiently for inference within the appraisal workflow.
Evaluation #
Performance Metrics #
Internal validation compares SCA recommendations to expert reviewer judgments using inter-rater reliability (IRR).
| Appraisal Framework | IRR with Experts |
|---|---|
| ROB2 | 0.60 |
| JBI | 0.52 |
| SIGN | 0.30 |
Overall performance varies depending on:
- Appraisal framework
- Study reporting quality
- The degree to which checklist criteria are explicitly stated in text
- The methodological complexity of the study under review
Performance Interpretation #
Agreement is strongest in frameworks where appraisal questions align more directly with explicit textual evidence. Lower agreement may occur in frameworks that rely more heavily on nuanced methodological interpretation or implicit reporting signals.
Known Issues #
- Some checklist items require contextual or domain-specific judgment beyond explicit statements in the article.
- PDF extraction quality may affect the model’s ability to identify relevant supporting evidence.
- Performance may vary across domains, study designs, and reporting standards.
- Some frameworks are more readily supported by current prompting and evidence extraction workflows than others.
Ethical Considerations #
Human-in-the-Loop Review #
Smart Critical Appraisal is designed as a decision-support system for methodological review.
The system recommends appraisal answers and highlights the passages that appear most relevant to each judgment, but final appraisal decisions remain under reviewer control. Human users may accept, revise, or clear AI-generated responses at any time.
This is particularly important because critical appraisal plays a central role in determining how much confidence reviewers place in a body of evidence. SCA is therefore built to support transparency and efficiency without removing human authority from the process.
Best practice is to review AI-generated answers alongside the annotations before finalizing appraisal records.
Limitations #
- Performance depends on the reporting quality of the full-text article.
- Poorly structured PDFs or extraction artifacts may reduce evidence quality.
- Some appraisal criteria are inherently interpretive and may not map cleanly to textual evidence alone.
- Current Smart CA compatibility is limited to supported appraisal frameworks.
- SCA only operates on included studies with full-text uploads available.
Planned Improvements #
- Expanded compatibility across additional appraisal frameworks
- Improved prompting and evidence attribution for nuanced methodological criteria
- Broader internal benchmarking across study designs and review types
- Enhanced reviewer tooling for auditing, revising, and validating AI-generated appraisal outputs
Contact Information #
For questions, feedback, or support, please contact support@nested-knowledge.com.
PALISADE Compliance #
Purpose #
To assist reviewers in conducting structured critical appraisal of included studies by generating checklist-based recommendations and evidence annotations from full-text articles.
Appropriateness #
SCA is appropriate for evidence synthesis workflows that require standardized methodological quality assessment, including systematic reviews, clinical evaluation reports, health technology assessments, and related regulatory or research deliverables.
It is not intended for clinical decision-making or diagnostic use.
Limitations #
- Outputs reflect LLM-based interpretation of study text in relation to structured appraisal questions.
- Reliability varies across frameworks and reporting styles.
- AI-generated answers should be reviewed by trained human assessors before final use.
Implementation #
Smart Critical Appraisal evaluates each checklist item by combining:
- The selected critical appraisal question
- The full-text content of the included study
- The structure of the configured appraisal system
For each question–study pair, the system uses a large language model to assess whether relevant evidence is present in the document and to recommend a structured appraisal response.
SCA also returns supporting annotations to indicate the text passages most relevant to each recommendation. These annotations are integrated directly into the platform’s appraisal workflow to support transparent review and revision.
The system is compatible with:
- JBI (2020)
- SIGN (2011)
- SIGN (2019)
- Cochrane RoB 2
Within the broader platform, manual critical appraisal supports additional systems including:
- Newcastle-Ottawa Scale
- ROBINS-I
- QUADAS-2
SCA does not overwrite human-completed responses and can be used in Single or Dual Critical Appraisal workflows.
Sensitivity and Specificity #
SCA is designed to support efficient and transparent appraisal rather than produce irreversible judgments.
Because critical appraisal often involves nuanced interpretation, the system is better understood in terms of agreement with expert reviewers than traditional classification metrics. Current agreement levels indicate moderate alignment in some frameworks, with variability depending on how directly the checklist can be inferred from text.
As with other LLM-assisted review tools, human oversight remains essential to ensure appropriate interpretation of methodology, bias, and study quality.
Algorithm Characteristics #
- Large Language Model–based checklist reasoning
- Per-question, per-document inference
- Full-text evidence review
- Annotation-based evidence highlighting
- Structured output within a predefined appraisal framework
Like ASTs, SCA is prompt-driven and semantic in nature, offering flexibility across complex appraisal questions while remaining probabilistic rather than deterministic.
Data Characteristics #
- Processes user-uploaded full-text PDFs for included studies
- Evaluates checklist questions against extracted article text
- Supports structured appraisal workflows inside the review platform
- Operates across heterogeneous biomedical and clinical research domains
User content is processed only for appraisal generation and is not retained beyond the relevant workflow.
Explainability #
SCA provides passage-level annotations to show the evidence supporting each recommended appraisal response.
These annotations help reviewers to:
- Trace recommendations back to the study text
- Validate or revise AI-generated answers
- Maintain an auditable appraisal process
- Preserve methodological transparency for publication, regulatory review, or internal quality assurance
Annotations are intended to support reviewer interpretation, not replace it.
Additional Notes on Compliance #
Smart Critical Appraisal is embedded directly within Nested Knowledge’s structured critical appraisal workflow. It is designed to accelerate one of the most time-intensive stages of systematic review while preserving rigor, auditability, and reviewer authority.
Because appraisal outcomes influence how evidence is interpreted downstream, expert oversight remains essential. SCA is therefore positioned as an assistive system that helps teams move faster without compromising the standards required for credible, publishable, and reviewable evidence synthesis.
