For enterprise level users only, Adaptive Smart Tags (ASTs) can be used to recommend or extract custom data from all study abstracts and/or full texts using an OpenAI Large Language Model (LLM). See a full disclosure of what data Adaptive Smart Tags use and how it works, see the Nested Knowledge AI Disclosure.
Adaptive Smart Tags gives you the option to:
- Guide Data Extraction of individual studies by highlighting in-text evidence to review
OR
- Auto-extract individual studies for immediate outputs and export
See below for instructions on how to set up Adaptive Smart Tags, optimize tag questions and generate evidence for review or immediate export.
Setting Up Adaptive Smart Tags #
1. Ensure Smart Tags is turned on #
You can control the use of Smart Tags, Core and Adaptive, in your nest. Note: Core Smart Tags is for pre-set questions (PICOs, Study Location, Type and Size) in your review, Adaptive Smart Tags is for custom questions.
The ability to use Smart Tags is turned on in Nest Settings by default but when this setting is off, the controls to generate ASTs and CSTs will be disabled and any tag recommendations will be hidden to reviewers.
2. Set up your Tag Hierarchy #
Since Adaptive Smart Tags use the tag questions you configure in the hierarchy to extract data from your studies, you need create some questions of interest relevant to your review. We recommend using Form-based Tagging mode for best results. View how to configure your tag hierarchy as well as our guidelines on how to optimise your questions.
There is no limit on the number of questions you may generate ASTs for but we do recommend having a comprehensive set configured before you generate ASTs, as to not decrease your generation limit too quickly!
3. Ensure your studies are ready for Extraction #
You’ll have the choice of generating ASTs for study abstracts or full texts, as well as the option to generate for included studies only. As a result, ensure you have met the below requirements:
For Abstracts
- Upload applicable literature and ensure all studies have an abstract by checking bibliographic data on each study or applying a bulk action to Sync Bibliographic data if missing.
For Full Texts
- Upload applicable literature and ensure all necessary studies have a full text, and studies are included as appropriate prior to generation.
4. Generate Adaptive Smart Tags #
When the above requirements are met, navigate to the Tag Hierarchy and select “Adaptive”. Here you can toggle your desired settings:
- Generate ASTs for all study Abstracts or Full Texts in your nest
- Generate for Included Records Only (this is checked automatically when Full Text is selected and is advised to avoid hitting the generation limit!)
- Refresh ASTs– only relevant after you have run ASTs previously. If you have made edits to the hierarchy or added new tags, select Refresh to clear the existing recommendations or applied tags and generate new recommendations or applied tags respectively. See more below on Refreshing ASTs.
You can then choose to Recommend or Apply.
- Selecting Recommend requires human input, generating recommendations for you to review in Tagging, alongside the question and you must accept or reject each suggestion
- Selecting Apply requires no human input, auto-extracting data without review in Tagging, and you may go straight to the outputs to view extracted data
Each nest has a limit of 5000 abstract generations and 1000 full text generations, allowing you to keep track of generations left each time you wish to re-run.
This currently takes around 1-5 minutes per study, so it can take some time for recommendations to be generated if you have a large number of included studies.
5. Review Recommendations #
If you select “Recommend”, the next step is to view these recommendations in Tagging.
Tag recommendations are displayed in the tab below Questions on the right-hand side (red). In Full Text view, it’s titled “Full Text Tag Recs” ; in Abstract view, it’s titled “Abstract Tag Recs”. Adaptive Smart Tags are distinguishable to Standard Tag Recommendations by a sparkle icon to the left (orange in below screenshot).
To view them in either the Abstract or Full Text views, the Questions tab (red) and the Tag Recs tab (blue) must both be open, and a question must be selected (light blue background) for recommendations to be displayed (orange). Select the recommendations to be taken to the corresponding area of the PDF (orange) where you may review the recommendation, edit and/or clear as appropriate before applying the tag.
6. View Auto-extracted data #
Explore from the Tag Hierarchy
From the Tag Hierarchy, you can view applied tag contents and studies reported at a glance. From the Configure Tag Hierarchy page, select the “Examine” icon (red). Then select a tag from the hierarchy or begin typing in the text box (blue). Excerpts will be listed for you to explore and make edits.
You’ll recognise this format from Study Inspector, and this tool allows you to “View in Inspector” (bottom right), which is a shortcut to Study Inspector where the tag filter is auto-applied. This can be a helpful workflow for reviewing applied tags.
Explore Outputs
Whether your generated ASTs as applied tags or recommendations, you can view the extracted data immediately via Qualitative Synthesis, download the complete spreadsheets via Study Inspector and/or build out custom updatable tables ready for download in Dashboard.
Refreshing Adaptive Smart Tags #
If you make changes or add more tags to the hierarchy after previously generating ASTs, or you’ve uploaded more studies to your nest you may wish to re-run them. To do so, simply navigate back to the Configure Tag Hierarchy page, select “Adaptive” and select the appropriate settings. See below for guidance on when to check or leave unchecked the “Refresh” box.
Check the Refresh box if:
- You’ve edited existing or added new questions to the tag hierarchy
- Selecting Recommend or Apply are separate actions; i.e. refreshing recommendations does not clear existing applied tags and refreshing applied tags does not clear existing recommendations
Leave the Refresh box unchecked if:
- The tag hierarchy is unchanged and new studies have been uploaded to the nest
- In the case you have edited the hierarchy and new studies have been uploaded, leaving the refresh box unchecked will only generate ASTs for the new questions for the new studies and the old ASTs for existing studies will remain. If you wish to generate ASTs for all new questions on all studies, select Refresh.
Note: you must check Refresh to generate new ASTs in any case where desired studies have pre-existing ASTs.
Clearing Adaptive Smart Tags #
After running ASTs, you may wish to clear the recommendations generated or the auto-applied tags to start from scratch. To do this, navigate back to the Configure Tag Hierarchy page, select “Adaptive” and toggle the “Clear” tab at the top of the modal instead of “Generate”.
This action will clear all ASTs generated from all abstracts and full texts permanently. You may choose to select if you’d like to clear recommendations or ASTs-applied tags from studies. Note: if you choose to clear recommendations, this will not clear the tags you as the user have applied based on these recommendations. If you choose to clear applied tags, these are only the tags that Adaptive Smart Tags has applied when “Apply” was selected in the Generate tab.
Internal Validation Testing of Adaptive Smart Tags #
Based on an internal validation using expert tagging, Adaptive Smart Tags offers an expected recall of 80% and precision of 70%. Given the customizability of tags and the distinct content in different reviews, recall and accuracy of Adaptive Smart Tags (for both Abstracts and Full Texts) will vary by project, driving home the need for expert confirmation of any recommendation.
Tag Recommendations in Screening Module #
Tag recommendations are also displayed in the Screening module and abide by the same rules. Ensure the Questions tab is open, a question is selected, and the Tag Recs tab is open to display recommendations (red). This often means closing the Screening tab for space purposes (blue).
