PICO Hierarchy Generator

Model Name: PICO Hierarchy Generator

Version: 1.0

---

Overview #

The Hierarchy Generator creates concept hierarchies from multiple research abstracts by extracting PICO entities and organizing them into meaningful relationships.

The workflow involves:

1. Running a PICO extractor on research abstracts to identify key entities.
2. Performing a frequency analysis to determine the most relevant entities
3. Using a LLM to analyze and organize these entities into hierarchical structures, with significant pre- and post-processing for consistency and relevance.

These hierarchies help users explore study characteristics, enabling better comprehension and synthesis of research evidence.

---

Intended Use #

• Primary Purpose: Assist researchers in organizing study features into structured hierarchies to facilitate evidence synthesis and systematic reviews.
• Intended Users: Researchers, healthcare analysts, and systematic review teams.

---

Evaluation #

• The tool’s output is assessed subjectively, based on whether the generated hierarchy is well-aligned to the research question and domain  or aligns with user expectations.
• Effectiveness depends on the quality of input abstracts and the relevance of extracted entities.

---

Ethical Considerations #

• Transparency: Hierarchies are traceable to source text via abstract annotations, though the internal reasoning of LLMs in constructing the hierarchy is opaque.
• Bias: Outputs may reflect biases in the LLM, the PICO extraction process, or the selection of input data.
• Human Oversight: Users must validate and adapt hierarchies for their specific research contexts.

---

Limitations #

1. Subjectivity: No formal evaluation metrics are available; users assess hierarchy quality based on perceived relevance and utility.
2. Explainability: While PICO extraction is traceable, the LLM step is a black box method and thus the reasoning cannot be understood.
3. Scope: The tool is currently optimized for English-language abstracts and PICO frameworks. Outputs may be less useful for unstructured or poorly described studies.

---

Planned Improvements #

1. User Feedback Loop: Incorporate mechanisms to learn from user preferences and improve hierarchy generation over time.
2. Explainability Features: Enhance post-processing to provide justifications for hierarchical relationships.
3. Multilingual Support: Extend support to non-English abstracts.

---

Contact Information #

For questions, feedback, or support, please contact support@nested-knowledge.com..

---

#

PALISADE Compliance #

Purpose

The Hierarchy Generator organizes extracted entities from multiple abstracts into hierarchical structures, facilitating evidence extraction and synthesis.

Appropriateness

The tool is appropriate for biomedical systematic review because its supervised portions are trained on biomedical entities, uses the broadly accepted PICO framework, and generates structures well-understood in the biomedical field as taxonomies, which facilitate grouping of biomedical evidence at various levels of precision.

Limitations

• Output quality depends on the clarity and consistency of the input abstracts.
• The lack of formal evaluation metrics makes output validation reliant on user judgment.
• Limitations of the data: Restricted to English-language abstracts; performance may vary for ambiguous or poorly structured abstracts.

Implementation

The tool combines PICO-based entity extraction with LLM-assisted hierarchical organization. Pre- and post-processing ensure that outputs are interpretable and relevant. As the tool makes use of OpenAI LLMs, it requires web access.

Sensitivity and Specificity

Not applicable; this is not a classification task.

Algorithm Characteristics

• Design: PICO entity extraction followed by frequency filtering, and LLM-based hierarchy generation.

Data Characteristics

• Training and Evaluation Dataset: Not traditionally trained; relies on pre-built PICO extraction and GPT-4o capabilities.

Explainability

While the PICO extraction step is fully interpretable, the GPT-4o step lacks transparency. Users can validate outputs against the source text.

---

Additional Notes on Compliance #

Hierarchies are generated per session, ensuring that sensitive data remains private and secure. The Hierarchy Generator tool securely shares necessary data with OpenAI to process requests. Per DPA, OpenAI does not store this data, and it is not used to train OpenAI’s models.