There are two Tag Recommendation modes:
- Standard Tag Recommendations use keyword searches and is available to all users
- Smart Tag Recommendations uses OpenAI, a Large Language Model that can read and answer questions, but is only available to enterprise customers.
See below for a brief overview of each type.
Standard Tag Recommendations #
Nested Knowledge automatically generates Standard Tag Recommendations for every included study. These are available for Full Texts (by default) but you may also turn on Abstract Tag Recommendations in Settings.
Tag recommendations are displayed in the tab below Questions on the right-hand side (red). In Full Text view, it’s titled “Full Text Tag Recs” (always displayed); in Abstract view, it’s titled “Abstract Tag Recs” (off by default, must be toggled on in Nest Settings).
Standard Tag Recs: Keyword Search #
Once you have uploaded the Full Text, standard tag recommendations will become available. To view them in either the Abstract or Full Text views, the Questions tab (red) and the Tag Recs tab (blue) must both be open, and a question must be selected (light blue background) for recommendations to be displayed (orange). Select the recommendations to be taken to the corresponding area of the text (orange) where you may accept or reject the recommendation before applying the tag.
Standard Tag Recs perform an automatic search by exact keyword look up across the whole full text or abstract. Learn more about Standard Tag Recs.
Smart Tag Recommendations #
For enterprise level users only, Smart Tag Recommendations can be used to generate recommendations found in study abstracts and full texts using OpenAI’s GPT-4 large language model. Below is a brief overview of the steps required to generate Smart Tag Recs, but see here for an in-depth walkthrough.
Additionally, see a full disclosure of what data Smart Tag Recommendations use and how it works, see the Nested Knowledge AI Disclosure.
1. Meet the requirements #
Unlike Standard Tag Recs, Smart Tag Recs do not load automatically. They must be intentionally turned on and are limited on regenerations. Therefore, they require a few requirements before turning them on:
- Set up your tag hierarchy with all questions of interest
- Ensure you have included all relevant studies*
- Upload full texts to all relevant studies*
* Unnecessary if Smart Tag Recs are only being run for Abstracts for all studies.
2. Turn on Smart Tag Recs #
Once the requirements are met, head to Nest Settings to switch on Smart Tag Recommendations (red). Be sure to toggle on Abstract Tag Recommendations if you wish to include these too, they can be run on the first 500 studies in the nest (blue). Smart Tag Recs are generated for included full texts only by default.
3. View and Apply Smart Tag Recs for each Question #
This currently takes around 1-2 minutes per study, so it can take some time for recommendations to be generated if you have a large number of included studies. It will also automatically update when new studies are added to the queue. Regenerations should only be used if the questions are altered.
Tag recommendations are displayed in the tab below Questions on the right-hand side (red). In Full Text view, it’s titled “Full Text Tag Recs” (always displayed); in Abstract view, it’s titled “Abstract Tag Recs” (off by default, must be toggled on in Nest Settings). Smart Tag Recommendations are distinguishable by a sparkle icon to the left (orange in below screenshot).
To view them in either the Abstract or Full Text views, the Questions tab (red) and the Tag Recs tab (blue) must both be open, and a question must be selected (light blue background) for recommendations to be displayed (orange). Select the recommendations to be taken to the corresponding area of the text (orange) where you may accept or reject the recommendation before applying the tag. Recs can also be applied in bulk.
4. Regenerating Smart Tag Recommendations and Error Reporting #
Recs are automatically generated as new records come into and/or are included in your nest. Only if you make changes to your tag hierarchy or switch tagging modes after turning on Smart Tag Recommendations, should you update the recommendations accordingly. Due to expense of generating smart tag recommendations, a limited number of regenerations are allowed in a nest at this time. This limit is 1 regeneration and can be used in Nest Settings–> “Regenerate Recommendations”.
Since the use of OpenAI for tag recommendations is in its beta phase, meaning it is technically feature complete but still in its early stages, there is a possibility you may run into errors. Be sure to let us know if this occurs.
Learn more about using Smart Tag Recommendations. Once finished with Tagging, take a look at Next Steps.
Internal Validation Testing of Smart Tag Recommendations #
In an internal validation, when compared against expert tagging, Smart Tag Recommendations had recall of approximately 50%-60% across three diverse Systematic Literature Review topics when employing OpenAI’s GPT 4 Turbo to tag Abstracts from underlying records. Given the customizability of tags and the distinct content in different reviews, recall and accuracy of Smart Tag Recommendations (for both Abstracts and Full Texts) will vary by project, driving home the need for expert confirmation of any Smart Tag Recommendation.
Tag Recommendations in Screening Module #
Tag recommendations are also displayed in the Screening module and abide by the same rules. Ensure the Questions tab is open, a question is selected, and the Tag Recs tab is open to display recommendations (red). This often means closing the Screening tab for space purposes (blue).
