For enterprise level users only, Smart Tag Recommendations can be used to generate text recommendations found in study abstracts and full texts using OpenAI’s GPT-4 large language model. It is non-generative AI and directs you to text excerpts within your record’s literature to provide evidence of data. See a full disclosure of what data Smart Tag Recommendations use and how it works, see the Nested Knowledge AI Disclosure.
Smart Tag Recommendations:
- Guide Data Extraction of individual studies by highlighting in-text evidence
- May be bulk applied to view rapid overview of AI-extracted data of interest
- May be generated for up to 250 included full texts and the first 1000 abstracts (from all records not just included) in your nest
See below for instructions on the requirements for Smart Tag Recommendations, how to set up optimized questions, turning on recs, and viewing them in individual studies or applying them in bulk for quick viewing & export.
1. Meet the requirements #
Unlike Standard Tag Recs, Smart Tag Recs do not load automatically. They must be intentionally turned on and are limited on regenerations. Therefore, they require a few requirements before turning them on:
- Set up your tag hierarchy with all questions of interest
- Ensure you have included all relevant studies so that they are in the Tagging queue*
- Upload full texts to all relevant studies*
* Unnecessary if Smart Tag Recs are only being run for Abstracts.
2. Turn on Smart Tag Recs #
Once the requirements are met, head to Nest Settings to switch on Smart Tag Recommendations (red). Be sure to toggle on Abstract Tag Recommendations if you wish to include these too, they can be run on the first 500 studies in the nest (blue). Smart Tag Recs are generated for included full texts only by default.
Upon selecting Smart Tag Recs, you will be shown a progress bar modal. Feel free to click out of the modal, and this will continue to run in the background. To check in on progress, select the Background Jobs icon (three horizontal bars in the top right, next to AskAI).
3. View and Apply Smart Tag Recs #
This currently takes around 1-2 minutes per study, so it can take some time for recommendations to be generated if you have a large number of included studies. It will also automatically update when new studies are added to the queue. Regenerations should only be used if the questions are altered.
Applying Smart Tag Recs to Individual Studies #
Tag recommendations are displayed in the tab below Questions on the right-hand side (red). In Full Text view, it’s titled “Full Text Tag Recs” (always displayed); in Abstract view, it’s titled “Abstract Tag Recs” (off by default, must be toggled on in Nest Settings). Smart Tag Recommendations are distinguishable by a sparkle icon to the left (orange in below screenshot).
To view them in either the Abstract or Full Text views, the Questions tab (red) and the Tag Recs tab (blue) must both be open, and a question must be selected (light blue background) for recommendations to be displayed (orange). Select the recommendations to be taken to the corresponding area of the text (orange) where you may accept or reject the recommendation before applying the tag.
For reference, the view in Abstract looks similar but works the same. When selected, recs will show up in blue. Since the abstract is shorter than the full text, there is no auto-scrolling and amongst RoboPICO & My Keywords can be tougher to see. In that case it can be best to toggle these off temporarily:
You can access Questions and Tag Recs in various places in your nest: Screening module, Tagging module, in Abstract or Full Text view, and Study Inspector.
Applying Smart Tag Recs in Bulk #
While applying smart tag recs by going through individual questions within individual studies can work best for project types such as systematic literature reviews, for other project types, such as scoping or rapid reviews, it can be helpful to apply all Smart Tag Recs in Bulk and take a look at the data as a whole.
Note: before applying smart tag recs in bulk, it’s important to note AI-extracted data without human oversight will not produce the same accuracy as with human oversight, so keep this in mind when reviewing the data afterwards!
Navigate to Study Inspector and Add a Filter
To apply smart tag recs in bulk, navigate to Study Inspector. As with all Bulk Actions, you must delete any existing filters and add a filter to display only your records of interest (red) where you can confirm the total number (blue). This is so you’re sure the action is being applied to these records and not all of them. There are various filters, but the most commonly used in this scenario is filtering to Included studies as below. See all our Study Inspector filters here.
Apply the Tag Recommendations Bulk Action
After filtering, select “Bulk Actions” in the top right (red), select “Apply Tag Recommendations” (blue) in the dropdown and “Apply”. This applies all the generated Smart Tag Recommendations for the filtered studies.
4. Regenerating Smart Tag Recommendations and Error Reporting #
Recs are automatically generated as new records come into and/or are included in your nest. Only if you make changes to your tag hierarchy or switch tagging modes after turning on Smart Tag Recommendations, should you update the recommendations accordingly. Due to expense of generating smart tag recommendations, a limited number of regenerations are allowed in a nest at this time. This limit is 1 regeneration and can be used in Nest Settings–> “Regenerate Recommendations”.
Since the use of OpenAI for tag recommendations is in its beta phase, meaning it is technically feature complete but still in its early stages, there is a possibility you may run into errors. Be sure to let us know if this occurs.
Learn more about using Smart Tag Recommendations. Once finished with Tagging, take a look at Next Steps.
5. View Outputs #
Applying tags, with or without Smart Tag Recommendations, generates Nested Knowledge Outputs. Whether you’ve applied them individually or in bulk, you may wish to view Qualitative Synthesis, create a Dashboard of extracted data, or download all extracted AI data.
Internal Validation Testing of Smart Tag Recommendations #
In an internal validation, when compared against expert tagging, Smart Tag Recommendations had recall of approximately 50%-60% across three diverse Systematic Literature Review topics when employing OpenAI’s GPT 4 Turbo to tag Abstracts from underlying records. Given the customizability of tags and the distinct content in different reviews, recall and accuracy of Smart Tag Recommendations (for both Abstracts and Full Texts) will vary by project, driving home the need for expert confirmation of any Smart Tag Recommendation.
Tag Recommendations in Screening Module #
Tag recommendations are also displayed in the Screening module and abide by the same rules. Ensure the Questions tab is open, a question is selected, and the Tag Recs tab is open to display recommendations (red). This often means closing the Screening tab for space purposes (blue).
