Model Name: Smart Study Type
Version: 1.0
Overview #
Smart Study Type is a supervised machine learning model designed to classify research studies into one of the following categories:
- Clinical
- RCT
- Observational
- Case report
- Review
- Preclinical
- Other
The model processes abstracts and titles and predicts the most specific study type for each input. It is intended to assist users by automating study classification in systematic literature reviews (SLRs).
Intended Use #
- Primary Purpose: Automatically classify study types for systematic literature reviews and similar applications.
- Intended Users: Researchers, healthcare analysts, and systematic review teams.
Training Data #
- Dataset: 140,000 abstracts scraped from PubMed, labeled by study type.
- Validation Dataset: A test set of 1,000 randomly sampled PubMed studies coded by experts.
- Data Limitations:
- The model was trained exclusively on English-language abstracts.
- The abstract labels were inferred, which may lead to errors.
Evaluation #
- Performance Metrics:
- Exact Match Accuracy: 74% on the validation dataset.
- Weighted F1-Score: 0.74.
- Recall for RCTs: 0.96.
Ethical Considerations #
- Human-in-the-Loop Limitations: Users are expected to review predictions. However, this is not an interactive human-in-the-loop system that prompts validation for individual classifications. Due to automation bias, users may rely on the model’s predictions without adequate scrutiny, which should be mitigated through training and awareness.
- Bias: The datasets were gathered from a random sample of pubmed abstracts, and therefore should not represent a biased sample.
Limitations #
- Exclusively trained and evaluated on English-language abstracts.
- Struggles to classify ambiguous or less specific study types, such as “Other.”
- Labels do not cover all possible study types.
- The model just gives a study type with no explanation.
Planned Improvements #
- Modify data gathering process to improve accuracy of labels
- Allow for more types of study design to be identified.
- Efforts to refine feature engineering and the classification algorithm to enhance overall accuracy.
- Adding calibrated classification probabilities to better communicate the confidence of the predictions and assist users in decision-making.
- Extend capabilities to non-English abstracts to increase global applicability.
Contact Information #
For questions, feedback, or support, please contact support@nested-knowledge.com.
PALISADE Compliance #
Purpose
The purpose of Smart Study Type is clearly defined: classify study types from research publications to assist in systematic literature reviews. Its implementation aligns with fair and ethical practices, as it operates within an oversight framework where users are encouraged to validate outputs.
Appropriateness
The tool is appropriate for detecting study types from research abstracts because it uses a random forest model, a machine learning technique well-suited for classification tasks involving high-dimensional and complex textual data. The model was trained on a dataset of 140,000 abstracts, enabling it to learn and generalize from a diverse and representative set of examples across different study types. Random forests are particularly appropriate for this task because they are robust to overfitting
Limitations
- Struggles with ambiguous categories like “Other.”
- Labels do not cover a wide range of study types.
- Lack of transparency
- Limitations of the data: Restricted to English-language abstracts; performance may vary for ambiguous or poorly structured abstracts. The labels were inferred, rather than known.
Implementation
The model is made easily available in cloud software and may run on standard hardware or GPUs for faster computation.
Sensitivity and Specificity
The model’s overall weighted F1-score is 0.74, with a high recall (0.96) for randomized controlled trials (RCTs), indicating strong sensitivity for this critical study type.
Algorithm Characteristics
- Design: TF-IDF, keyword detection features to XGBoost multi-label classification.
- Transparency: The model’s architecture and performance metrics are documented for reproducibility.
Data Characteristics
- Training Data: 140,000 PubMed abstracts labeled by study type.
- Evaluation Data: 1,000 PubMed studies coded by experts.
Explainability
The model outputs the most specific predicted study type, enabling straightforward human interpretation and validation. However, it does not produce a rationale for the decision.
Additional Notes on Compliance #
This algorithm is not trained on the input data. It does not learn, store, or transmit input data, ensuring privacy and avoiding potential biases or ethical concerns related to training data.
