Critical appraisal of non-randomized studies of interventions: ROBINS-I tool

A Summary of

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355; i4919.

Relevance for Public Health

The ROBINS-I tool can be applied to any public health research question that examines the intervention effect of a public health intervention. This includes a number of observational study designs such as cohort studies, quasi-randomized trials and other concurrently controlled studies, as well as case-control studies, cross-sectional studies, interrupted time series and controlled before-after studies.

Description

The ROBINS-I tool and manual (Risk Of Bias In Non-randomized Studies - of Interventions) can be used to evaluate the risk of bias in non-randomized studies of interventions (NRSI) that compare the health effects of two or more interventions.

The ROBINS-I tool is an update to the previous ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions). The tool is based on the Cochrane "risk of bias" (RoB) tool for randomized trials, and also builds on related tools such as QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies).

The ROBINS-I is a tool to rate risk of biases in a non-randomized study of interventions. It outlines seven domains where biases might occur. The first two domains of bias occur in the "pre-intervention" phase, one domain of bias occurs in the "at intervention" phase and four domains of bias occur in the "post-intervention" phase. The specific bias domains are as follows:

Pre-Intervention

  • Bias due to confounding
  • Bias in selection of participants into the study

At Intervention

  • Bias in classification of interventions

Post-Intervention

  • Bias due to deviations from intended interventions
  • Bias due to missing data
  • Bias in measurement of outcomes
  • Bias in selection of the reported result

The tool also offers a number of explanations for ratings, terms, definitions and theory.

Accessing the Method/Tool


Language(s)

English

Format(s)

On-line Access

Cost

None

Implementing the Method/Tool


Time for Participation/Completion

1 to 2 hours

This tool requires an intense review of study methodology. With an increasing number of studies, the time to implement this tool increases.

Additional Resources and/or Skills Needed for Implementation

The ease with which this tool can be used coincides with knowledge and theory of study design.

Steps for Using Method/Tool

The users begin by outlining signalling questions, explanations and response options for each domain of bias. The typical response options are listed below, with notable exceptions outlined in the manual:

  • Yes
  • Probably yes
  • Probably no
  • No
  • No information

Then, users judge each domain to determine if it falls within a risk of bias category, including:

  • Low risk of bias
  • Moderate risk of bias
  • Serious risk of bias
  • Critical risk of bias

The tool then provides an approach to map the RoB judgements within domains to a single RoB judgement across domains for an outcome. In addition, the tool provides a framework for reaching an overall RoB judgement across multiple outcomes in a study.

Therefore, the overall use of ROBINS-I includes:

  • RoB judgements for signalling questions
  • RoB judgements within domains
  • RoB judgements across domains for one outcome
  • RoB judgements across domains for more than one outcome

Who is involved

Anyone who reads public health research.

Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Evaluation and Measurement Characteristics


Evaluation

Has been evaluated.

The tool was piloted with a working group that implemented the use of the tool across a number of topic areas using NRSI, and the revisions needed were addressed. Further pilot testing was conducted and feedback was collected with the target audience of first-time users using a telephone interview format. The tool was then launched in 2014 in a workshop format and user feedback from this event and additional training events in 2015 lead to final tool refinement.

Validity

Not applicable

Reliability

Not applicable

Methodological Rating

Not applicable

Method/Tool Development


Developer(s)

Cochrane Methods Bias
Website: http://www.riskofbias.info/

Method of Development

The development of the tool is described in detail in the corresponding BMJ publication (Supplemental materials). In brief, the tool was developed over three years, largely by expert consensus, and following the seven principles previously described for assessing risk of bias in clinical trials.

Release Date

2016

Contact Person/Source

JAC Sterne
School of Social and Community Medicine
University of Bristol
Bristol, UK BS8 2PS
Email: jonathan.sterne@bristol.ac.uk

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Resources

Title of Primary Resource The risk of bias in non-randomized studies - of interventions (ROBINS-I) assessment tool
File Attachment None
Web-link
Reference

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355; i4919.

Type of Material Tool
Format On-line Access
Cost to Access None.
Language English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Title of Supplementary Resource ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions
File Attachment None
Web-link
Reference

Sterne, J.A.C., Hernan, M.A., Reeves, B.C., Savovic, J., Berkman, N.D., Viswanathan, M, et al. (2016). ROBINS-I: a tool for assessing riskof bias in non-randomised studies of interventions. BMJ, 355:i4919.

Type of Material Journal
Format Periodical
Cost to Access None.
Language English
Conditions for Use

© BMJ Publishing Group Ltd 2016

Title of Supplementary Resource Risk of bias in non-randomized studies of interventions (ROBINS-I): Detailed guidance
File Attachment None
Web-link
Reference

Sterne JAC, Higgins JPT, Elbers RG, Reeves BC and the development group for ROBINS-I. Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance, updated 12 October 2016. Available from http://www.riskofbias.info [accessed February 7, 2017].

Type of Material Manual
Format On-line Access
Cost to Access None.
Language English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Have you used this resource? Share your story!