Announcing ‘shukra’: Bringing NMA to the Web

Announcing ‘shukra’: Bringing NMA to the Web
Karl Holub
Ischemic Stroke Recanalization Treatment Network (Network Meta-Analysis)

This week, we published our most recent open source project, ‘shukra’, to npm. shukra is a NodeJS module for performing Network Meta-Analysis. It aims to provide interpretable results efficiently (in less than 10 milliseconds) for simple integration into web applications.

About Network Meta-Analysis

Network Meta-Analysis (NMA) is a statistical method for research synthesis. Given a collection of multi-arm studies on a set of treatments, NMA produces treatment effect estimates for all pairs of treatments. Consider the following set of studies & treatment arms:

  • Study 1
    • Treatment A
    • Treatment B
  • Study 2
    • Treatment A
    • Treatment B
    • Treatment C
  • Study 3
    • Treatment B
    • Treatment C
  • Study 4
    • Treatment C
    • Treatment D

There are several complexities that make a standard aggregative meta-analysis difficult. Treatment D is only directly compared to C, meaning we can’t directly estimate D’s efficacy relative to A & B. Additionally, Study 2 has 3 arms which we need to account for.

NMA alleviates these issues by:

  • Breaking all studies into treatment pairs of direct evidence
    • e.g. Study 2 is synthesized as 3 different pairs of treatments: A vs. B, A vs. C, and B vs. C
  • Computing “indirect” evidence using cross-study contrasts
    • e.g. We have no direct B vs. D contrasts, but we can use all B vs. C & C vs. D direct evidence to form indirect B vs. D evidence.
    • e.g. If we know Lisa is 6 inches taller than Mark, and Mark is 1 inch taller Johnny, then we can indirectly infer that Lisa is 7 inches taller than Johnny (despite the fact we never directly measured their height difference).

There is some statistical complexity behind these steps, for example:

  • Variance of indirect evidence must be corrected (upwards), since we are combining two varying quantities
  • Indirect evidence should be made arbitrarily “deep”, utilizing effects estimates across the entire treatment network.
    • e.g. In the B vs. D case, we could additionally use A vs. C & A vs. B evidence to  supplement the B vs. D estimate

However, a user typically doesn’t need to worry about these implementation details. The paradigm is simple: make all treatments pairwise and compute direct and indirect evidence. This approach works for all types of outcomes- continuous (e.g. mean differences) & discrete (e.g. odds or risk ratios), and can even be made to handle multivariate outcomes.

NMA at Nested Knowledge

Here’s an example NMA, applied to our Ischemic Stroke dataset, looking at post-treatment recanalization (a binomial outcome). On top is a treatment network; each vertex is a treatment, and the thickness of each edge indicates how many studies directly contrasted the two connected treatments. On the bottom are the estimated odds ratios of successful recanalization for all thrombectomy devices against a control (thrombolysis).

Ischemic Stroke Recanalization Treatment Network (Network Meta-Analysis)

Ischemic Stroke Thrombectomy Recanalization NMA

 

shukra

shukra is a NodeJS tool for performing NMA. It is designed to run server side and produce a fitted model in less than 10 milliseconds. Because code is more precise than English, below is an example usage on a sample of our Ischemic Stroke dataset:

const {fixedEffectsOddsRatioNMA} = require('shukra');

const studyPMIDs = [22913726, 22913726, 25671797, 25671797, 30994831, 30994831, 30994831, 22932715, 22932715];
const treatments = ['Trevo', 'Solitaire', 'IV-tPA', 'Solitaire', 'Trevo', 'Solitaire', 'IV-tPA', 'Solitaire', 'Merci'];
const recanalizationCounts = [10, 12, 15, 33, 22, 26, 16, 52, 26];
const armSize = [13, 20, 35, 35, 30, 32, 27, 58, 55];

const nma = fixedEffectsOddsRatioNMA(studyPMIDs, treatments, recanalizationCounts, armSize);

nma.getEffect('Trevo', 'IV-tPA');
// 3.51
nma.getEffect('Solitaire', 'Merci');
// 8.99
nma.computeInferentialStatistics('Solitaire', 'IV-tPA');
/**
 * {
 *   p: 0.00044400977525627106,
 *   lowerBound: 1.978461619896957,
 *   upperBound: 11.080007730824889
 * }
 */
nma.getEffect('Merci', 'IV-tPA');
// 0.52

Notice that, as is the case for Merci vs. IV-tPA, even when the published body of literature does not contain a direct contrast of two treatments, shukra provides an effect estimate.

Why Open Source?

At NK, we use shukra on almost every web request to produce inferential statistics for Nests. On account of the NodeJS ecosystem having no open source implementations of NMA, we had to implement it ourselves. Why would we make such an integral part of our platform publicly available?

We made the decision on two accounts:

  • Transparency: We document all our methodologies, and published code is the most transparent methods documentation we can provide.
  • Pervasiveness: We want to grow the cause of web-based, living systematic review. If publishing parts of our tooling helps others to grow the cause, we view it as a success for everyone.

Features Coming Soon

Features you can expect to see in shukra (and on nested-knowledge.com!) soon are:

  • Random Effects models – capturing study heterogeneity in the NMA, e.g. due to differing patient populations
  • Power analysis – what is the statistical power of the meta-analysis?
  • Single arm studies – how can we integrate single arm studies into the NMA to increase power?