Model Name: Adaptive Smart Tags (AST) #
Version: 1.0 #
Overview #
Adaptive Smart Tags (AST) is a large-language-model–powered information extraction and recommendation system designed to support systematic literature reviews by identifying custom, user-defined concepts within study abstracts and/or full texts.
Unlike Core Smart Tags (which address predefined fields such as PICO or study type), AST enables reviewers to configure arbitrary tagging questions via a hierarchical tag structure. For each study and tag, AST can either:
- Recommend extracted values for human review, or
- Apply tags automatically for immediate downstream analysis and export.
AST also provides in-text annotations highlighting evidence used to support each extracted concept, enabling transparent human validation.
The system is intended for enterprise-level workflows requiring scalable, customizable data extraction across large evidence corpora.
Intended Use #
- Primary Purpose:
To automate or accelerate extraction of custom review variables from biomedical abstracts and full texts, supporting qualitative synthesis, structured outputs, and evidence inspection. - Intended Users:
Systematic review teams, clinical researchers, health economists, and enterprise evidence synthesis groups. - Limitations:
- Outputs depend heavily on the clarity and completeness of source documents.
- Performance varies by tag formulation and domain.
- Not designed to replace expert judgment; human confirmation is strongly recommended, especially for inclusion-critical variables.
Training Data #
- Dataset:
AST relies on a general-purpose large language model provided by OpenAI. The underlying model is trained on a mixture of licensed data, data created by human trainers, and publicly available text. - Validation Dataset:
Internal benchmarking datasets derived from expert-tagged abstracts and full texts across multiple nests and tagging configurations (Form-Based Tagging and Standard Tagging). - Language:
Primarily English.
AST does not perform fine-tuning on customer data. User content is processed transiently for inference only.
Evaluation #
- Performance Metrics
Internal validation using expert tagging shows the following representative performance:
| Dataset | Recall | Precision | Annotation Rate | Timing (s) |
|---|---|---|---|---|
| Abstract – Form-Based Tagging | 0.85 | 0.69 | 96% | 123 |
| Full Text – Form-Based Tagging | 0.83 | 0.67 | 94% | 420 |
| Abstract – Standard Tagging | 0.80 | 0.59 | Unknown | — |
Overall expected performance:
- Recall: ~80–85%
- Precision: ~67–70%
Results vary substantially depending on:
- Question design
- Tag hierarchy structure
- Document type (abstract vs full text)
- Subject matter
- Known Issues:
- Precision may decrease when tag questions are underspecified.
- Over-application of tags can occur when hierarchies lack sufficient contextual constraints.
- Certain datasets were only partially QAed, meaning reported precision likely underestimates true model capability.
- Full-text extraction is significantly slower than abstract-only processing.
Ethical Considerations #
- Human-in-the-Loop Limitations: AST supports both recommendation-based workflows (with expert review) and fully automated extraction. While auto-apply enables rapid outputs, it increases risk of silent errors.
Best practice is to use Recommend mode for critical variables and to validate extracted concepts using the provided annotations.
Limitations #
- Performance depends on how well tag questions are written.
- Ambiguous or implicit concepts in source text may be missed or misinterpreted.
- Precision is lower for loosely defined tags.
- Generation limits apply (per nest) for abstracts and full texts.
- Full-text processing is slower and more resource-intensive.
Planned Improvements #
- Improved guidance and tooling for tag question optimization.
- Expanded internal QA datasets, especially for full-text workflows.
- Enhanced evidence attribution and summarization options.
- Continued performance tuning for precision on complex or abstract concepts.
Contact Information #
For questions, feedback, or support, please contact support@nested-knowledge.com.
PALISADE Compliance #
Purpose #
To extract or recommend structured, review-specific data from unstructured biomedical literature at scale, reducing manual workload while maintaining human oversight.
Appropriateness #
AST is appropriate for evidence synthesis and research workflows requiring customizable variable extraction. It is not intended for clinical decision-making or diagnostic use.
Limitations #
- Outputs depend on LLM interpretation of text and user-defined prompts.
- Recall and precision vary by project.
- Automated application increases the likelihood of unreviewed errors.
Implementation #
AST uses a large language model to evaluate each configured tag against each study, effectively asking:
“Does this concept occur in this document?”
For every tag–study pair, the model returns:
- A recommended or applied value
- Optional supporting annotations pointing to relevant text spans
Users may operate AST in:
- Recommend mode (human review required), or
- Apply mode (automatic extraction)
Annotations are enabled by default to support traceability.
Sensitivity and Specificity #
- Higher sensitivity (recall) than precision by design, favoring concept discovery over conservative extraction.
- Precision improves with well-scoped, form-based tag questions.
- Full-text tagging generally shows slightly lower precision and longer processing times than abstract-only tagging.
Algorithm Characteristics #
- Large Language Model–based semantic extraction
- Per-tag, per-document inference
- Optional evidence highlighting
- Supports both assisted and fully automated workflows
Unlike deterministic classifiers (e.g., SST), AST is probabilistic and prompt-driven, reflecting the flexible but less predictable nature of LLM-based systems.
Data Characteristics #
- Processes user-provided abstracts and/or full texts
- No persistent storage of inputs beyond inference
- Supports arbitrary custom tag hierarchies
- Operates across heterogeneous biomedical domains
Explainability #
AST provides sentence- or passage-level annotations identifying text used to justify each extracted concept. These highlights enable reviewers to:
- Verify correctness
- Edit or reject recommendations
- Build trust in automated outputs
Annotations are intended as supporting evidence, not definitive explanations, and should be interpreted alongside expert judgment.
Additional Notes on Compliance #
Adaptive Smart Tags processes user content only for prediction and extraction. Data is not retained beyond the generation workflow. Due to variability inherent in large language models, expert confirmation remains essential. AST is designed as a decision-support system to enhance — not replace — human review.
