Critical appraisal of non-randomized studies of interventions: ROBINS-I tool

A summary of

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355; i4919.

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2017). Critical appraisal of non-randomized studies of interventions: ROBINS-I tool. Hamilton, ON: McMaster University. (Updated 01 September, 2017) Retrieved from http://www.nccmt.ca/knowledge-repositories/search/281.

Categories: Tool, Appraise

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health

The ROBINS-I tool can be applied to any public health research question that examines the intervention effect of a public health intervention. This includes a number of observational study designs such as cohort studies, quasi-randomized trials and other concurrently controlled studies, as well as case-control studies, cross-sectional studies, interrupted time series and controlled before-after studies.

Description

The ROBINS-I tool and manual (Risk Of Bias In Non-randomized Studies - of Interventions) can be used to evaluate the risk of bias in non-randomized studies of interventions (NRSI) that compare the health effects of two or more interventions.

The ROBINS-I tool is an update to the previous ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions). The tool is based on the Cochrane "risk of bias" (RoB) tool for randomized trials, and also builds on related tools such as QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies).

The ROBINS-I is a tool to rate risk of biases in a non-randomized study of interventions. It outlines seven domains where biases might occur. The first two domains of bias occur in the "pre-intervention" phase, one domain of bias occurs in the "at intervention" phase and four domains of bias occur in the "post-intervention" phase. The specific bias domains are as follows:

Pre-Intervention

  • Bias due to confounding
  • Bias in selection of participants into the study

At Intervention

  • Bias in classification of interventions

Post-Intervention

  • Bias due to deviations from intended interventions
  • Bias due to missing data
  • Bias in measurement of outcomes
  • Bias in selection of the reported result

The tool also offers a number of explanations for ratings, terms, definitions and theory.

Implementing the Method/Tool
Steps for Using Method/Tool

The users begin by outlining signalling questions, explanations and response options for each domain of bias. The typical response options are listed below, with notable exceptions outlined in the manual:

  • Yes
  • Probably yes
  • Probably no
  • No
  • No information

Then, users judge each domain to determine if it falls within a risk of bias category, including:

  • Low risk of bias
  • Moderate risk of bias
  • Serious risk of bias
  • Critical risk of bias

The tool then provides an approach to map the RoB judgements within domains to a single RoB judgement across domains for an outcome. In addition, the tool provides a framework for reaching an overall RoB judgement across multiple outcomes in a study.

Therefore, the overall use of ROBINS-I includes:

  • RoB judgements for signalling questions
  • RoB judgements within domains
  • RoB judgements across domains for one outcome
  • RoB judgements across domains for more than one outcome
Who is involved

Anyone who reads public health research.

Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Evaluation and Measurement Characteristics
Evaluation
Has been evaluated.

The tool was piloted with a working group that implemented the use of the tool across a number of topic areas using NRSI, and the revisions needed were addressed. Further pilot testing was conducted and feedback was collected with the target audience of first-time users using a telephone interview format. The tool was then launched in 2014 in a workshop format and user feedback from this event and additional training events in 2015 lead to final tool refinement.

Validity
Not applicable
Reliability
Not applicable
Methodological Rating
Not applicable
Method/Tool Development
Developer(s)

Cochrane Methods Bias
Website: http://www.riskofbias.info/

Method of Development

The development of the tool is described in detail in the corresponding BMJ publication (Supplemental materials). In brief, the tool was developed over three years, largely by expert consensus, and following the seven principles previously described for assessing risk of bias in clinical trials.

Release Date
2016
Contact Person/Source

JAC Sterne
School of Social and Community Medicine
University of Bristol
Bristol, UK BS8 2PS
Email: jonathan.sterne@bristol.ac.uk

Resources

Title of Primary Resource
The risk of bias in non-randomized studies - of interventions (ROBINS-I) assessment tool
File Attachment
None
Web-link
Reference

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355; i4919.

Type of Material
Tool
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License


Title of Supplementary Resource
ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions
File Attachment
None
Web-link
Reference

Sterne, J.A.C., Hernan, M.A., Reeves, B.C., Savovic, J., Berkman, N.D., Viswanathan, M, et al. (2016). ROBINS-I: a tool for assessing riskof bias in non-randomised studies of interventions. BMJ, 355:i4919.

Type of Material
Journal
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

© BMJ Publishing Group Ltd 2016


Title of Supplementary Resource
Risk of bias in non-randomized studies of interventions (ROBINS-I): Detailed guidance
File Attachment
None
Web-link
Reference

Sterne JAC, Higgins JPT, Elbers RG, Reeves BC and the development group for ROBINS-I. Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance, updated 12 October 2016. Available from http://www.riskofbias.info [accessed February 7, 2017].

Type of Material
Manual
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License


Have you used this resource? Share your story!