Overview #

The first step in quality control is to understand the research question and proposal if available. Once the project goals are clear, identify the aim of the QC by asking the following:

• Is the main goal of this QC to finalize the data before statistical analysis?

In the case that the project is unclear and the research design needs to be refined, one of the goals of QCing will be to refine the research question. Evaluate studies for common themes while reviewing the data extraction with the following guiding questions:

• Does the research design make sense and is the aim of the research clear?
• Are the primary outcomes (or the data elements that are of highest interest) clear and reported in a standard, measurable way?
• Are there common outcomes or interventions mentioned in the included studies that are missing?

Quality Assessment of articles in Nested Knowledge is performed in Study Inspector.

Screening #

When reviewing the extraction for each study, assess if the study has any of the following detriments to inclusion:

• Low data density or quality (does not have a significant amount of data extraction or does not incude the main outcomes of interest)
• Subpopulation that could skew results in a certain direction (ex: children, vulnerable population, elderly)
• The cohort is not homogenous (ex: all are considered COVID-19 patients but some patients are suspected and some are PCR-confirmed positive)
• Study type (consider if lower level evidence such as case reports or case series should be included for the project)
• Does not report primary outcome of interest (ex: reports 30-day mortality when the primary outcome of interest is 90-day mortality)

In the case that the study should not be included, note the reason that exclusion is recommended for secondary review.

When performing an SR/MA, you should also be sure that it does not have an excessively small or excessively large number of included studies. Always seek input from a statistician on this matter, as an “appropriate size” for an SR/MA is highly variable and depends on the specific study design and client requests. Generally speaking, you might expect a meta-analysis with inferential statistics to include anywhere from 4 to 50 studies.

Extraction #

Examine the data elements for extraction. There may be inconsistencies that occurred as a result of variances in study reporting that can only be uncovered after reviewing the extraction of several studies. When examining the extraction for each study, look out for the following common data element errors:

• Clarify any data elements that are vague (ex: how are we defining clinical improvement?) and make sure that the collected data is consistent with that definition.
• Studies that have a limited amount of data elements extracted (low data density). Comment directly on the study to be evaluated for secondary review.
• If you notice that a data element is being reported more often as a mean instead of a median, this may need to be changed.
• Check that outcomes are reported on consistent scales with the same units (ex: if pain is reported on a VAS vs NRS scale, then there is a need to create two separate data elements).
• Check units of data elements to ensure they are consistently reported the same way.
• Make sure that the tagged intervention groups match the extracted intervention groups.

Go through each study and check that each element, intervention, and follow-up period are correctly extracted, paying special attention to the abstract, tables, methods and results.

Common extraction errors to correct:

• Elements that are collected as outcomes do not have loss to follow up reflected in total.
• Combined means were not combined correctly
• Continuous data elements are being reported using the incorrect statistical measure (standard error is not the same as standard deviation)
• Make sure that elements are being extracted at the same time point or that the wording of the study is consistent with what is being collected (ex: 30-day mortality is not the same as 90-day mortality). Clarify if unsure of what the time point of interest is. Look out for inconsistencies across studies and note if there are timepoints that are commonly reported but are missed.

Statistical Errors and Clarifications #

QC requires a thorough understanding of the statistical measurements that are encountered when reading scientific literature. Refer to the following list if clarification is needed.

Mean	The average of a set of numbers. This measure can be combined across populations if there are two means reported. This is NOT the same thing as Least Square (LS) mean.
Median	The middle value of a listed set of numbers. This measure can not be combined or used as a mean.
Standard Deviation (SD)	A measure of the amount of variance within a set of numbers. Typically reported with a mean value using the symbol ‘±’
Standard Error (SE)	The standard deviation of a sample of the larger population. This is NOT the same thing as SD and should not be used in replacement of SD.
Range	The difference between the highest and lowest numbers in a given set. Typically given alongside the median.
Interquartile Range (IQR)	The difference between the upper and lower quartiles. This is NOT the same thing as Range and should not be used as a replacement.

NK-specific QC #

To be done in addition to steps outlined above. qc_checklist_for_wikis.docx

Tagging #

QC after review Screening section and review before Extraction section.

• Look at the sunburst diagram in qualitative synthesis and evaluate gap teeth. Gap teeth occur when a study has been tagged at the higher level instead of the lower, most specific level. If studies are tagged with Study Design instead of the most specific tag, this may result in gap teeth. Gap teeth can also result when there is a small amount of studies tagged.
• For tagging only nests, evaluate each study for tagging similar to the process of checking the extraction.
• Look at quantitative synthesis and note data elements that are “empty” or very small in the drop down. If there are less than 5 studies with data extracted for that element, it is worth proposing eliminating that data element.

Extraction #

• Look at quantitative synthesis and note data elements that are “empty” or very small in the drop down. If there are less than 5 studies with data extracted for that element, it is worth proposing eliminating that data element.

Note any thoughts or concerns you have in the comments section of the study. If you have done extensive revisions for the study, note the problem with the first extraction and how you fixed it. Justify any recommendations for inclusion/exclusion.

Final Steps #

• Let the team or project lead know that QC is complete and include notes and recommendations in the email.
• In the Nest Comments, type up a QC report detailing any changes that have made to the nest and any clarifications or new standards that have been set. Recommendations should not be included in the QC report, only actions that were taken. Include the date that QC was completed.