Appraising the risk of bias in randomized trials using the Cochrane Risk of Bias Tool

A summary of

Higgins, J.P.T., Sterne, J.A.C., Savović, J., Page, M.J., Hróbjartsson, A., Boutron, I., Reeves, B., Eldridge, S. (2016). A revised tool for assessing risk of bias in randomized trials. In: Chandler, J., McKenzie, J., Boutron, I., Welch, V. (editors). Cochrane Methods. Cochrane Database of Systematic Reviews, 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2017). Appraising the risk of bias in randomized trials using the Cochrane Risk of Bias Tool. Hamilton, ON: McMaster University. (Updated 01 September, 2017) Retrieved from http://www.nccmt.ca/knowledge-repositories/search/280.

Categories: Tool, Appraise

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health

The tool was originally developed with clinical trials and their biases in mind. However, the tool is applicable in the assessment of study quality from any published randomized trial, including those conducted with a public health perspective.

Description

A risk of bias tool was created to assess the potential for bias in randomized trials included in systematic reviews and meta-analyses. In 2005, the Cochrane Collaboration decided to improve upon the existing tools at the time, mainly scales and checklists, with and without scoring respectively. The original risk of bias (RoB) tool was launched in 2008, and its most current update, version 2.0, is summarized here.

The RoB tool 2.0 provides a framework for considering the risk of bias in the findings of any type of randomized trial. The current tool (Oct 20, 2016) is based on the potential sources of biases that may arise in clinical trials.

The RoB tool 2.0 is for use with individual randomized trials, randomized parallel group trials, cluster-randomized trials and randomized cross-over trials.

The following five domains are to be used in the assessment of the risk of bias in individual randomized trials:

  1. Bias arising from the randomization process
  2. Bias due to deviations from intended interventions
  3. Bias due to missing outcome data
  4. Bias in measurement of the outcome
  5. Bias in selection of the reported result
Implementing the Method/Tool
Steps for Using Method/Tool

The RoB tool 2.0 involves the user making a series of judgements about the extent of bias arising from the conduct of the trial. The tool focuses on the risk of bias rather than reporting characteristics. The tool thus allows readers of the review to decide whether they agree or not with the judgements made in assessing the risk of bias for trials.

A key feature is the inclusion of signalling questions within each domain of bias. These questions are used to help make judgements about the risk of bias. The response options for signalling questions are:

  • Yes
  • Probably yes
  • Probably no
  • No
  • No information

The signalling questions should be answered independently from one another, after which domain-level judgements about the risk of bias are determined. The risk of bias judgements are:

  • Low risk of bias
  • Some concerns
  • High risk of bias

The key is consistency, ensuring domain-level judgements about risk of bias mean the same across the five domains and thus constitute a similar view on the trustworthiness of the trial conduct. The five domains include:

  1. Bias arising from the randomization process
  2. Bias due to deviations from intended interventions
  3. Bias due to missing outcome data
  4. Bias in measurement of the outcome
  5. Bias in selection of the reported result

Finally, an overall risk of bias judgement can be made for a specific outcome for a given trial, using the following qualifiers:

  • Low risk of bias: the study is judged to be at low risk of bias for all domains for this result.
  • Some concerns: the study is judged to have some concerns of bias in at least one domain for this result.
  • High risk of bias: the study is judged to be at high risk of bias in at least one domain for this result OR the study is judged to have some concerns for multiple domains in a way that substantially lowers confidence in the result.

In summary, the steps to using the tool include:

  • Determining the risk of bias in each of the five domains
  • Determining the risk of bias across all five domains for a specific outcome

Additional features of the tool are further described in the Manual (See Supplementary Resources). The developer's website also provides slightly modified tools depending on the trial design. The tool reviewed here (primary resource) is for individually randomized, parallel group trials.

Who is involved

Systematic reviewers use the tool, but systematic review readers can also see the judgements made by the reviewers.

Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Evaluation and Measurement Characteristics
Evaluation
Has been evaluated.

The original tool has been evaluated (see Supplemental Resources, Higgins et al., 2011).

Validity
Validity properties meet accepted standards.
Reliability
Reliability properties meet accepted standards.

Reliability has not been extensively studied (BMJ), however some items are more difficult to assess by users. Ongoing guidance and training materials have been developed to help improve reliability.

Methodological Rating
Not applicable
Method/Tool Development
Developer(s)

MRC Network of Hubs for Trials Methodology Research
Website: http://www.riskofbias.info/

Method of Development

In 2005, a group of experts, including statisticians, epidemiologists and others, developed the tool and used informal consensus to refine it. Specific criteria for the tool were developed by this group, and further refinements were made after pilot work of the tool at meetings and workshops. The first version of the tool was launched in 2008, with refinements made in 2016.

Release Date
2016
Contact Person/Source

Julian PT Higgins
MRC Biostatistics Unit
Institute of Public Health
Cambridge, UK CB2 0SR
Email: julian.higgins@mrc-bsu.cam.ac.uk.

Resources

Title of Primary Resource
The RoB 2.0 tool (individually randomized, parallel group trials)
File Attachment
None
Web-link
Reference

Higgins, J.P.T., Sterne, J.A.C., Savović, J., Page, M.J., Hróbjartsson, A., Boutron, I., Reeves, B., Eldridge, S. (2016). A revised tool for assessing risk of bias in randomized trials. In: Chandler, J., McKenzie, J., Boutron, I., Welch, V. (editors). Cochrane Methods. Cochrane Database of Systematic Reviews, 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.

Type of Material
Online tool
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 International License


Title of Supplementary Resource
Revised Cochrane risk of bias tool for randomized trials (RoB 2.0)
File Attachment
None
Web-link
Reference
Type of Material
Manual
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

© 2016 by the authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License


Title of Supplementary Resource
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
File Attachment
None
Web-link
Reference

Higgins, P.T.J., Altman, D.G., Gotzsche, P.C., Juni, P., Moher, D., Oxman, A.D., et al. (2011). The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 343, d5928.

Type of Material
Journal
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

© 2011 Creative Commons


Have you used this resource? Share your story!