If you are familiar with Tagging in Nested Knowledge, you know how integral the process of setting up a tagging hierarchy is to overall study design. If Tagging is new to you, check out this article and then come back to this one! In short, Tagging is the critical function for both building hierarchies of the concepts of interest and extracting them from underlying studies, and until recently, our AI automations could help with the extraction step–but all hierarchy-building was entirely up to you. Core Smart Tags fixes this by automatically generating your starter hierarchy based on the studies in your nest, as well as adding the ability to drill down on key study characteristics, from Search Exploration to Dashboard on Synthesis.
In this article, we’ll go over how Core Smart Tags work, where they add functionality to Nested Knowledge, as well as share some preliminary accuracy statistics so that you can get a sense for how you might use them to augment your review.
When we set out to create Core Smart Tags, we had a few objectives in mind. First, we wanted to provide greater configuration assistance than is provided with our existing Custom Smart Tag Recommendations, which help automate as much of the data extraction process as possible, but which depend on your careful configuration. Second, we wanted to improve the speed and accuracy of these features, with more consistent and replicable hierarchy structures. Last but not least, we wanted to ensure that whatever we built could be integrated into the existing product in a way that expanded understanding, both for recommending content extraction and for visualizing the final outputs.
In order to accomplish each of these key objectives, the automated hierarchy building and concept extraction could not cover all possible topics; we needed to narrow in on the question of “what concepts will consistently be of interest to your research topics?” Additionally, these commonly extracted elements had to be found consistently in a study’s abstract or bibliographic information, so that the hierarchy can be generated as soon as a Search or Import is added. Lastly, these concepts needed to be structured so that we can take them from unstructured text and create an organized hierarchy. For this reason, the Core Smart Tags that can be be generated in Nested Knowledge are:
To tackle this task, we eventually settled on a combination of supervised machine learning models, entity recognition models, and heuristics to achieve the kind of speed, accuracy and cost-effectiveness required.
These models enable AutoLit to process dozens of records per second with an impressive ~80% accuracy (more on accuracy later). With this newfound efficiency, your review can start with immediate categorization of all studies based on these key concepts, and you can use this either to assist with improving your search and screening process, or to bolster your Tagging process by extracting Core Smart Tags alongside Custom Smart Tag Recommendations. In short, the advantages of Core Smart Tags are:
Because Core Smart Tags do not rely on the full text, we can use the underlying models in Search Exploration to get a more complete picture of the studies in a preliminary search strategy even before it is executed. In addition to PICOs, we have added study size, location, and type to Search Exploration, allowing you to get a bird’s eye view of study contents prior to importing records. For example: if your research question focuses on a specific geography, you’ll be able to rapidly scope the size of your review & approximate screening time. Similarly, if your research question requires specific levels of evidence or study designs, Study Type & Size directs you to amend your search strategy based on the information provided by Core Smart Tags.
To make the best use of Core Smart Tags for your preliminary search, enter your key terms into the Concepts in the upper left of the module, and use Groups to put similar concepts together (e.g., group all of your Interventions so they are all searched as a category). Then, Refresh Exploration, and Core Smart Tags will recommend your PICOs, Location, Type, and Size! Add more terms to further develop your search, and click on PICOs of interest to see further definitions, and study types and locations to see study lists. When finished, Finalize your search and run it on your database of interest!
Once a search has been finalized and records have been imported into your nest, Core Smart Tags switch from being your Search strategy tool to providing structure and time savings to your data extraction. Core Smart Tags are automatically generated for each study upon search import and can be added to your tag hierarchy in the Tagging Configuration page by clicking the “Core” button.
Core Smart Tags can be used as recommendations or they can be applied as tags directly to studies. If you are completing a systematic literature review, it is likely best to populate these AI extractions as recommendations so that you can check them in each study, but for a Rapid AI Review, automatic application can give you immediate outputs from your tagging without curation.
This means that Core Smart Tags can be used to fully automate the extraction of key concepts for review types where speed is important and accuracy of ~80% is acceptable. That said, Core Smart Tags may also be used where the goal is to produce a truly systematic review, because complete human oversight is available when enabling Core Smart Tags as recommendations.
Note: if you’re not familiar with Custom Smart Tag Recommendations, which offer a completely custom, though more computationally intensive tag recommendation, check out this article.
In effect, treat Core Smart Tags as a quicker way of building a preliminary hierarchy, which you can then augment with your own tags in order to achieve a full extraction. While simply adding children to the root of a Core Tag will not generate more recommendations or apply the tag to your studies, running Custom Smart Tag Recommendations will give you custom recommendation for tags created after Core Smart Tags have been run. It’s worth noting too, that Core Smart Tags themselves will be excluded from Custom Smart Tag Recommendations.
Concurrently with the release of Core Smart Tags, we added three new visualization modes to tag card type in Dashboard on Synthesis. While these modes were specifically designed to complement the capabilities of Core Smart Tags, the Sunburst card can be used with any tag, and the Histogram card can be used with any tag with numeric contents.
Sunburst Card
The Sunburst mode allows you to replicate the sunburst diagram from Qualitative Synthesis, complete with filtering. By selecting a parent tag in the configuration step, each child tag will be organized into a sunburst diagram, allowing dashboard viewers to hover over a segment to view the number of studies associated.
Choropleth
This mode offers dashboard viewers an interactive color-coded study frequency map, whereby hovering over a specific geography, they can see the number studies associated with that location. In practice, the Choropleth map really only works properly when you select the study location tag while configuring this card type. This is because it uses the frequency of applications for the selected tag’s text options, and it’s looking for exact matches to an expected set of countries. That said, you can certainly manually apply the study location tag to a study if the Core Smart Tags fail to extract the correct study location to ensure high accuracy.
Histogram
While introduced to offer a way to display the distribution of study sizes, the histogram visual makes use of any tag with the numeric content type. The histogram is automatically binned and transformed according to properties of the data, so it will be visually reliable as you extract data in an ongoing manner.
PICOs
The primary powerhouse behind the PICO core tag is an open-source entity extraction model with an F1 score of 0.74 (SOTA at time of writing) on the EBM PICO dataset.
However, we are adding a bit more magic to create a hierarchy and context for the PICO terms once they are recognized; due to the qualitative nature of that process, we do not have a quantitative measure of the accuracy of that output. As such we will be relying on user feedback, in order to figure out ways to improve those results. If you notice anything funky or undesirable in your automatically generated PICO hierarchy, feel free to let us know, we’d love to hear from you.
Study Type
In the process of developing the Study Type Core Tag, we built a test data set composed of 1,000 randomly sampled studies from PubMed that were coded for study size by experts. Our supervised model achieved an exact match 74% of the time in this data set. In practice, we expect higher accuracy in SLR settings, where study types are less a random sample (the model biases towards clinical study types) and hierarchical accuracy is higher.
Study Size
Using the PICO Corpus dataset, our model achieved 91% accuracy. While we’re proud of the headline number, it’s worth noting that this dataset is composed entirely of RCTs. We expect our NER + heuristics approach to generalize well to other study types but performance may vary. It’s important to note that “size” is ambiguous for certain study types; for example, in a meta-analysis, the model will prefer the pooled number of patients when reported, else the number of studies.
Study Location
The Study Location Core Tag model achieves an overall accuracy of 78%, or an F1 of 0.8 on a random sample of PubMed records with NCT ID linkage. This accuracy statistic will be most reliable on RCT and cohort designs, which are most frequently registered. The definition of a “study location” will vary with study types. When in doubt, our model will attempt to provide the most sensible answer. The location reported will always be a country, but it may be the author’s location in the case of a narrative review, for example.
Other Details
Whether Core Smart Tags are applied directly to studies or are offered to reviewers as recommendations, each has slightly different characteristics. The PICO Core Tag and its children, for example, will always expect text contents, and will always be accompanied by an abstract annotation; depending on the contents of the study’s abstract, you may find many recommendations/applications for that study. The following table provides a breakdown of roughly what you can expect for each tag type:
Ongoing Testing and Backfill Timeline
While Core Smart Tags is still in beta, we are pleased to offer access to the feature to all users, though this may be subject to change as we evaluate costs and continue to develop new Core Tag types. Core Smart Tags are available now in any newly created nests, and we are working on backfilling older nests, so that existing nests can have these available in the next few months. If you are an existing customer looking for an expedited backfill, feel free to reach out.
Core Smart Tags are a novel artificial intelligence (AI) system in the Nested Knowledge workflow that can provide immediate assistance with your Search strategy, build your Tagging hierarchy automatically, and add more flexible visuals to your research outputs. They can be a powerful tool for extraction recommendations, or alternatively, can be used to build out Rapid AI Reviews to get an approximate breakdown of study contents within minutes. Lastly, Core Smart Tags have been tested against early internal test sets and are undergoing improvements, but we strongly encourage you to test them yourself.
Similarly, if you would like more information on how to best make use of Core Smart Tags for your next Nest, contact our Support team by filling out the form below, and we’ll meet with you one-on-one to help you get started quickly. Happy reviewing!
Yep, you read that right. We started making software for conducting systematic reviews because we like doing systematic reviews. And we bet you do too.
If you do, check out this featured post and come back often! We post all the time about best practices, new software features, and upcoming collaborations (that you can join!).
Better yet, subscribe to our blog, and get each new post straight to your inbox.
In the realm of conducting systematic literature reviews (SLRs), the question invariably arises: how do these new technologies compare to the trusty spreadsheet? Spreadsheets offer
