Lesson 2: Study Design

Kevin Kallmes: Hi, this is Kevin Kallmes presenting the second lesson in Nested Knowledge’s series on how to review the medical literature. However, before we jump in with that lesson on how to design the protocol for your review, let’s do a quick review of lesson one. In lesson one, we learned what is a systematic review and when should I do one? So a systematic review is effectively a tool for answering medical research questions by comparing and combining studies from the medical literature, and generally, that research question takes the structure of, “For population, P, how do interventions, I, perform as measured by outcomes, O?” We also went over at the highest level how to do a systematic review, and that general structure was you start with a search, so you run a Boolean query on a database like PubMed, and then you move forward and you screen those resulting articles to include the ones that are going to be relevant to your search and exclude the ones that are irrelevant. Then you want to tag any included article with content of interest, so qualitative concepts and extract quantitative data elements. And then finally, you’ll present both the qualitative information and the quantitative information you’ve gathered in synthesis.

KK: So in this lesson, we’re actually gonna be drilling down on how to plan out each of those steps by designing your study and drafting your protocol. I like to think about the protocol drafting process as answering a set of important questions about your review. So if you’ve done lesson one, you should be able to answer question one here very easily by just saying, “I’m trying to figure out the answer to my medical research question,” but you also need to build out the team of who is helping you with every step in that systematic review process. You also need to plan, not just to screen articles based on whether you think they’re relevant, but based on a set of pre-configured exclusion criteria, they will help you separate the wheat from the chaff. Then you also need to plan in advance what data you will be collecting in order to answer your research question.

KK: And this usually looks like some combination of characteristics of the underlying studies, characteristics of the underlying patient population, and then the interventions of interest and the outcomes of interest. Now, if you are wondering where you can actually draft this protocol, Nest have built-in protocols that are on the Nest homepage to help you and your collaborators distribute work and always come back to your main question as you’re doing your review. And if you’re part of an organization, your organization may have templates that you can use to put together your protocol, usually in some form of questions that you need to answer. To go through the details of how to draft a protocol, let’s use that same example that we used in lesson one of the Basilar artery stroke review that Nested Knowledge has published. As we recall, a Basilar artery stroke is a blood clot in the key artery that feeds your brain stem, meaning that it is a type of stroke that leads to really high morbidity, bad neurological outcomes and really high mortality. And that review is published both on our site and in the SPIN Journal.

KK: Our protocol starts with the research question which we crafted in lesson one. So as we recall, we are looking at patients with acute ischemic stroke of the Basilar artery, and we were comparing the endovascular therapy called thrombectomy against the medical therapy of thrombolytic or clot-busting drug. And we were comparing those therapies with respect to key patient outcomes like mortality and neurological outcome, and stroke physicians actually use a specific outcome scale for neurological outcome called the Modified Rankin Scale score. So research question is done.

KK: Let’s move on and identify our collaborators. The PI for our project was Dr. Jeremy Heit from Stanford, and his resident at the time, Gautam, also led a lot of the collection planning and writing. We also had a data gathering team and we thought it was important to bring on a biostatistician, which I strongly recommend as a part of your review team. Then we had to define the scope of our review. And as I said earlier, you cannot just include and exclude articles based on whether you think they’re relevant. You have to use consistent rules. And what those rules look like are a set of inclusion and exclusion criteria. So for a Basilar artery review, we want to look at clinical studies, so not in vitro, not in vivo or swine studies. We wanted to look at only human clinical studies reporting our patient outcomes of interest for our interventions of interest with respect to our patient population of interest.

KK: Now, that is generally a correct overview of what studies we want to include, but just because it’s correct, it doesn’t mean it’s complete. It’s also important to identify within this specific reasons why you might exclude a study that otherwise meets your inclusion criteria. For our review, we wanted to drill down only on prospectively collected, unbiased evidence. And so we actually had the exclusion criteria of retrospective studies. And you may ask, “How could you know whether you could exclude retrospective studies?” and the answer is, we looked into the literature before we began the review and made sure that we characterized the sets of study types that we were seeing there, so that we would be able to come in and plan this based on an understanding of where the literature stands as of today.

KK: And one of my biggest recommendations as part of protocol drafting is do some reading in the literature, make sure that you’re actually reflecting the underlying literature’s traits when you’re crafting both the scope and the data collection. So we also noted while we were becoming experts on the subject that thrombectomy was actually adopted as standard of care in 2015, so we saw that as a transition in standard of care, and therefore anything before 2015, we actually decided to exclude from our review. We also excluded any type of study that of course wouldn’t report the patient outcomes, interventions and populations of interest. And since we reviewed the underlying studies, we could also see that there were existing comparative studies specifically on Basilar artery strokes, and so to reduce bias, we narrowed our review on comparative evidence and excluded any study that didn’t have at least one thrombectomy and one thrombolysis study arm.

KK: We also noticed while we were reading up on the underlying literature that some studies would report Basilar artery strokes in the same population as a bunch of other stroke locations, and in that case, we are effectively at the mercy of the underlying authors. So if these studies mixed their stroke locations and didn’t report outcomes by location and by therapy, that meant that we weren’t able… We won’t be able to actually extract that evidence. And so we made the exclusion criteria where any study that does not separate outcome by stroke location must also be excluded. And then of course, we excluded studies not reporting each of our interventions of interest. And we kept going with these exclusion criteria until we identified all rules that we wanted to use to throw studies out of our Nest.

KK: So scope, to define scope in a study, you’re effectively going to be doing an extra exercise in creating inclusion and exclusion criteria that reflect the types of studies that you want in and the types of studies that you want to throw out. And it’s imperative that you base this on some understanding of what the literature contains, otherwise you might set yourself up with a review that has either an incredibly high number of biased results or possibly zero includable studies once you get your search in order. So you have your inclusion and exclusion criteria. You are done with your scoping exercise.

KK: Now, in terms of data collection, I actually think that it’d be really helpful for us to jump directly into the nest of interest. So I’m going to hop over and show you the hierarchy that we built in this nest. So here you can see, this is our Basilar artery nest. And in terms of study characteristics, what we thought was important from underlying studies was their country of origin and their study type. You could build this out with any other qualitative characteristic of a study that you want to collect simply by creating a new tag and housing it under the parent tag of study characteristics. But we wanted to drill down much more on the actual population of interest. So for that, we went to two sources, one was the underlying literature and the second was actually the NIH.

KK: So in the underlying literature, we found that people very commonly drilled down on the timing of stroke therapy. This is likely because stroke is an emergent condition where you get rapidly worse with every hour that passes. And so underlying studies generally reported multiple periods during the treatment process for every underlying patient. So we thought it was important to also reflect the time that it took from the admission, from the onset of the stroke to the admission to the hospital, to the beginning of therapy.

KK: We also went to the NIH, which puts out a list of common data elements or effectively evidence that should be reported in any study in stroke patients. So the National Institute for Neurological Disease and Stroke has put out guidelines that say that you should generally be collecting mean and median age as well as gender for underlying patients. So we added, again, both underlying literature-based and common data element-based data elements to our review. For intervention, very straightforwardly, we mapped out endovascular therapy and standard medical therapy, so thrombectomy and thrombolysis. And then for outcomes, we of course covered mortality and MRS, but we notice two things. First is that MRS wasn’t actually reported in a consistent manner. Some studies would report it as MRS zero to two, so patients that were between zero and two on the scale, but others would report it as MRS zero to three.

KK: So we basically needed to reflect whichever way the underlying study reported it. We also reported mortality, and then SICH, which is symptomatic intracranial hemorrhage, which can often lead to neurological deficit or mortality. And then lastly, the NINDS also recommended collecting NIHSS, which is a stroke-specific severity scale, both before and after treatment. So we collected NIHSS at baseline, and then after therapy. What does this look like? Well, in action, when you are done with your study, you get something like our qualitative synthesis which shows exactly those study design characteristics, population characteristics, interventions of interest and clinical outcomes of interest. And I’ll also note that we went on to also collect angiographic outcomes, so effectively, on imaging, was the clot cleared or not using a stroke-specific scale? 

KK: And for each of these, we are able to drill down by just clicking on the tag of interest and identifying the study we want to see, for instance, how does endovascular therapy perform with respect to mortality, and then clicking in on the underlying studies of interest. With that defined and with our hierarchy displayed, let’s hop back into the PowerPoint. Give me one sec. So with respect to data collection, we looked at the study characteristics of study type. So among prospective studies we tagged for RCTs, registries and prospective cohort studies, we tagged for location, we also collected patient characteristics from underlying studies and that were common data elements recommended by the NINDS, such as NIHSS score, age, gender, and race. Then we identified our interventions of interest and the… Which is relatively straightforward in our case, right? We only wanted to reflect thrombectomy versus thrombolysis. If we wanted to compare, for instance, specific endovascular devices, we would’ve needed to build out that hierarchy to reflect every specific device or if we wanted to look at every specific drug, we could have built out our hierarchy there.

KK: But in our case, since we wanted to do a basically therapy type level comparison, we’re done by just identifying thrombectomy and thrombolysis. And then for outcomes, we collected not only Modified Rankin Scale score and mortality, but also collected the NIHSS or stroke severity scale. We collected symptomatic intracranial hemorrhage, which is a complication of the procedure that can lead to bad outcomes. And then we also collected the clot clearance on angiography, so an imaging-based metric of therapeutic success.

KK: Also, as we were building that, we went over a couple of hierarchy hints and I have a couple more to add to it. So in building out that hierarchy, we do want to reflect the data collection, both the qualitative tags and the quantitative extraction that we’re going to do. And in doing so, we are reflecting our study design and those ever present P, I, and O that come from our research question. So if you don’t have something that roughly reflects P, I, and O in your hierarchy, you may not be reflecting your research question correctly. Also, simple is beautiful. You saw that hierarchy, nothing extra and nothing extraneous. When we previously built that hierarchy, we’ve often said, “Let’s collect whatever the underlying study reports and let’s get as detailed as we can.” And generally, I actually find that that can distract. The better part of finding is filtering.

KK: The better part of gathering is figuring out what not to gather. So in building your hierarchy, make sure that you’re drilling down only on the actual pieces of evidence that will help you answer your research question, not necessarily reflect every single data point that every underlying article presented. Then a Nested Knowledge-specific recommendation, make sure that your tagging is preparing you for extraction. So any concept that is both a qualitative and quantitative concept, you can actually configure as a tag and then set yourself up for extraction. And let me show you what this looks like in action if we jump back into our nest. Once we’re looking at a study, if we’ve tagged correctly, so if we’ve captured the underlying tags… Sorry. If we’ve captured the underlying content of the study through our tags, then as we are extracting, not only will the data elements that were tagged be prioritized in the list, but by clicking this little tag icon, I can auto-jump to the part of the PDF that actually reports the data element of interest, in this case, age.

KK: So if you’ve tagged and used the annotation correctly, you should be setting yourself up to extract a heck of a lot faster if you’re using Nested Knowledge. So one quick Nested Knowledge-specific recommendation, then let’s hop back in, and this is just reflecting exactly that. Then as you are actually building out your hierarchy, make sure that you’re not just reflecting what you originally said you wanted. Say for us, it was Modified Rankin Scale score, we ended up needing to reflect that as a dichotomous variable, so MRS zero to two, but in underlying studies, they didn’t always characterize it that way. So we needed to make sure that we described what that really means, so make sure we reflect the acronym correctly, and then also reflect synonyms such as good neurological outcome or good functional outcome, both of which reason underlying articles and would have been missed if we didn’t add a description and aliases reflecting acronyms and synonyms of our tags.

KK: And then lastly, and I said this for the scope, for both scoping your review and planning your extraction, it’s incredibly helpful if you’ve read some of the underlying literature. That will help you make sure that the study designs that you’re seeking actually exist. It will identify for you the patient characteristics, the interventions and the outcomes that are generally commonly reported, and it’ll basically help you build out in advance what you’re likely to be able to extract from the underlying studies. So don’t come in expecting the systematic review to teach you as you go the basic concepts that you need. What you should actually be doing is learning those basic concepts, finding them in the literature, and then building out your systematic review to reflect those pieces of it that help you answer your research question.

KK: In summary, in lesson two, we’ve learned about protocol building in our Basilar Artery review. Study design lives in your protocol in your Nest. It builds your research question out to cover the scope. Defining scope means identifying inclusion and exclusion criteria and prepares you for screening. It also prepares you for data collection through qualitative, so tagging, and quantitative extraction. And then you also need to make sure that you’re planning who is doing what within your review. Some themes that I’m sure you’re sensing from across both lessons, PIO or PICO are everywhere. Simplicity really helps clear communication, not only of your research question to your internal team, but of what you have demonstrated in your review to your eventual readers. We went over the basic idea that before you design your review, you should be doing some reading.

KK: So the systematic review process should look something like self-educate or read up on studies, design your study, and then execute your study, and you should constantly, constantly, constantly be going back to your research question to help you answer questions that arise along the way. Okay, so what can you do with this after lesson one? First, your protocol should be your central way that you communicate with your team. And if you’re doing it with Nested Knowledge, we have a comment field that’s right next to the protocol that you can use to update each other on your progress. You should be pre-writing the method section of your eventual publication effectively in your protocol.

KK: So if you do a good job in your protocol, the actual exercise of writing a publication, the method section should be very easy and should effectively pull the information that you put into your protocol. And I just wanted to presage that in future lesson, we’re going to be going through, not only how to design and execute on this in general, but also how to read a study to include or exclude it, to tag it, and then to extract data from it. And we’ll follow a study through its whole life cycle in a review. Looking forward to it. Thanks so much and have a great day.

[music]