Guidelines on Smart Tag Recommendations

Smart Tag Recommendations is most effective when the tag hierarchy is kept as simple and concrete as possible. See below for tips on how to optimise smart tag recommendations:

Configuring the Tag Hierarchy #

Mode

• Form-based Tagging mode (default) works best
  • Smart Tag Recs work in Standard mode too, but does not have explicit Questions to feed to the LLM

Tags

Ensure you provide:

• Tag Name
• Question Type (where relevant),
• Question and Description text to give details beyond the
• Tag Name,
• Aliases, if there are common synonyms or acronyms
• All of the above, as well as the general hierarchy structure, is fed to the LLM

Tag Name

• Limit to 4-5 words max, spell out acronyms, be careful with special characters ($, :, -, % etc.) only use if the term is commonly reported
• Ensure terms are spelled correctly but don’t worry about variations (e.g. UK vs US English)

Question type

• Use Single Apply where possible
• Use Select types if total answers/child tags are less than 8, too many tags can overwhelm the LLM
• Currently only one recommendation is generated per question, which can be less helpful for Multiple Select. Single Apply is more likely to identify area of paper containing various answers

Questions

• Keep it simple and direct, don’t add too much detail
• Present or past tense questions starting with “What” or “Where” tend to work best

Aliases (Optional)

• When the tag name cannot be identified (this includes both question and child tags for select question types), the LLM searches for aliases
• This is most helpful to provide for field-specific terms

Contents (Tag Tables and Options)

• Where possible, use free-text Tags; if you use a Tag Table, the LLM can scroll you to relevant text, but cannot assist with extraction
• Similarly, if employing Options, the LLM can scroll you to relevant text, but will not select Options on your behalf

Examples of Optimized Questions:

1. Tag Name: “Population”; Question Type: Single Apply, Question: “What population was studied?
2. Tag Name: “Location”; Question Type: Single Apply, Question: “Where did the study take place?”
3. Tag Name: “Number of Participants”; Question Type: Single Apply, Question: “What was the total number of participants?”
4. Tag Name: “Study Design”; Question Type: Single Select, Question: “What is the study design?” Child Tags: Randomized Controlled Trial, Active Trial, Prospective Study
5. Tag Name: “Clinical Outcomes”; Question Type: Multiple Select, Question: “What clinical outcomes are reported?” Child Tags: Mortality at 90D, Neurological Outcome (mRS 0-2 at 90D), Symptomatic Intracranial Hemorrhage (sICH)

Note: only one answer will be generated!Edit

Turning on Smart Tag Recs #

Only after your tag hierarchy is configured (your studies are included and full texts uploaded for full text generation) should you toggle on Smart Tag Recs:

In Settings → Tag Recommendations → Choose Type: Smart. By default, Smart Tag Recs searches Full Texts but you may also wish to turn on Abstract Tag Recommendations, which may be toggled on in the same section of Settings. The LLM will then also highlight evidence on the Abstract page.

Learn more about meeting the requirements and turning on Smart Tag Recs.

Recommendation Limits #

Abstracts: Smart Tag Recs is currently able to run on 1,000 total abstracts or less, and runs on all abstracts in your nest

Full Texts: Smart Tag Recs is currently able to run on 250 full texts or less, and runs only on included reports.Edit

Using Smart Tag Recs #

Smart Tag Recs can take some time to generate. Feel free to click out of the modal and let it run in the background, you can check back in on its progress anytime using the icon.

As applicable, Abstract or Full Text Tag Recs are displayed as a right hand tab (blue) wherever you can view a record: Screening & Tagging modules and Study Inspector. Ensure you have Questions open (red), with a question selected for the Smart Tag Recommendation to be displayed (light blue background). Clicking the rec will auto-scroll to its suggested text excerpt within the Abstract/Full Text for you to review (orange) and apply or reject. If a recommendation cannot be found, there will be none listed.

Smart Tag Recs display a sparkle icon to distinguish from Standard Tag Recs.

Advanced: Bulk Apply Smart Tag Recs #

While we encourage users to oversee the LLM by reviewing and either accepting or rejecting each recommendation, you may want to apply all the LLM recommendations at once. For instance, when completing a scoping review, you may want to view all LLM recommendations without any curation.

In short, head to Study Inspector, filter to your studies of choice, select Bulk Actions → Apply Tag Recommendations, Smart Tag Recs and Full Text are selected by default, select Apply. This will apply all Tag Recommendations, which you can then view in Qualitative Synthesis. Learn more here.

Bulk Actions is an advanced tool, make sure to familiarize yourself with the use of Bulk Actions and Study Inspector first before going ahead with this action!

FAQs #

What data does OpenAI GPT 4 use? Are there restrictions in place on the use of my data?

When using Smart Tag Recommendations, your tag hierarchy and all relevant text (abstract and full text) content are sent to OpenAI via their API. OpenAI is blinded to your identity, and processes this information for the express purpose of providing Smart Tag Recommendations, per standard Data Processing Addendum. Your data will not be used to inform or train OpenAI products.

I don’t see any recommendations. What am I doing wrong?

First, make sure recommendations were successfully generated in Background Jobs. Then make sure you are on either the Abstract or Full Text page, you have the right hand Questions and Tag Recs tabs open and a Question is selected. Try going through a few questions in the form, it may be that fewer recommendations were generated than expected. Still nothing? You may need to simplify your hierarchy and regenerate recs. If that doesn’t work, contact support@nested-knowledge.com.

The recommendation is highlighting the same excerpt for multiple questions or isn’t applicable to my question, can I retrain the AI?

OpenAI GPT 4 works to find the most succinct answer to the question, which means it often highlights the abstract/similar excerpts. Currently there is no way to retrain the AI but we are working on having multiple recommendations generated for you to choose from… coming soon!

I added new tags to the hierarchy and new Smart Tag Recs were not generated, is it automatic?

No, after configuring new questions, Smart Tag Recs must be regenerated in Settings. Note: each nest is limited to one regeneration due to LLM costs. If you need to regenerate Smart Tag Recs after using your one regeneration, contact support@nested-knowledge.com.