Critical appraisal of non-randomized studies of interventions: ROBINS-I tool

A Summary of

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355; i4919.

Description

The ROBINS-I tool and manual (Risk Of Bias In Non-randomized Studies - of Interventions) can be used to evaluate the risk of bias in non-randomized studies of interventions (NRSI) that compare the health effects of two or more interventions.

The ROBINS-I tool is an update to the previous ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions). The tool is based on the Cochrane "risk of bias" (RoB) tool for randomized trials, and also builds on related tools such as QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies).

The ROBINS-I is a tool to rate risk of biases in a non-randomized study of interventions. It outlines seven domains where biases might occur. The first two domains of bias occur in the "pre-intervention" phase, one domain of bias occurs in the "at intervention" phase and four domains of bias occur in the "post-intervention" phase. The specific bias domains are as follows:

Pre-Intervention

  • Bias due to confounding
  • Bias in selection of participants into the study

At Intervention

  • Bias in classification of interventions

Post-Intervention

  • Bias due to deviations from intended interventions
  • Bias due to missing data
  • Bias in measurement of outcomes
  • Bias in selection of the reported result

The tool also offers a number of explanations for ratings, terms, definitions and theory.

Access the Tool

Format of the Tool


Language(s)

English

Format(s)

On-line Access

Cost

None

Using the Tool


Time for Participation/Completion

1 to 2 hours

This tool requires an intense review of study methodology. With an increasing number of studies, the time to implement this tool increases.

Additional Resources and/or Skills Needed for Implementation

The ease with which this tool can be used coincides with knowledge and theory of study design.

Steps for Using Method/Tool

The users begin by outlining signalling questions, explanations and response options for each domain of bias. The typical response options are listed below, with notable exceptions outlined in the manual:

  • Yes
  • Probably yes
  • Probably no
  • No
  • No information

Then, users judge each domain to determine if it falls within a risk of bias category, including:

  • Low risk of bias
  • Moderate risk of bias
  • Serious risk of bias
  • Critical risk of bias

The tool then provides an approach to map the RoB judgements within domains to a single RoB judgement across domains for an outcome. In addition, the tool provides a framework for reaching an overall RoB judgement across multiple outcomes in a study.

Therefore, the overall use of ROBINS-I includes:

  • RoB judgements for signalling questions
  • RoB judgements within domains
  • RoB judgements across domains for one outcome
  • RoB judgements across domains for more than one outcome

Who is involved

Anyone who reads public health research.

Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Evaluation of the Tool


Evaluation

Has been evaluated.

The tool was piloted with a working group that implemented the use of the tool across a number of topic areas using NRSI, and the revisions needed were addressed. Further pilot testing was conducted and feedback was collected with the target audience of first-time users using a telephone interview format. The tool was then launched in 2014 in a workshop format and user feedback from this event and additional training events in 2015 lead to final tool refinement.

Validity

Not applicable

Reliability

Not applicable

Methodological Rating

Not applicable

Development of the Tool


Developer(s)

Cochrane Methods Bias
Website: http://www.riskofbias.info/

Method of Development

The development of the tool is described in detail in the corresponding BMJ publication (Supplemental materials). In brief, the tool was developed over three years, largely by expert consensus, and following the seven principles previously described for assessing risk of bias in clinical trials.

Release Date

2016

Contact Person/Source

JAC Sterne
School of Social and Community Medicine
University of Bristol
Bristol, UK BS8 2PS
Email: jonathan.sterne@bristol.ac.uk

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Have you used this resource? Share your story!