Appraising Qualitative, Quantitative, and Mixed Methods Studies included in Mixed Studies Reviews: The MMAT

A summary of

Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., Boardman, F., Gagnon, M.P., & Rousseau, M.C. (2011). Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Retrieved from http://mixedmethodsappraisaltoolpublic.pbworks.com. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2015). Appraising Qualitative, Quantitative, and Mixed Methods Studies included in Mixed Studies Reviews: The MMAT. Hamilton, ON: McMaster University. (Updated 22 June, 2017) Retrieved from http://www.nccmt.ca/resources/search/232%20(accessed%20May%202017).

Categories: Tool, Appraise

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health

Criteria for the MMAT were developed based on a thematic analysis of the quality appraisal processes that were conducted in 17 health-related systematic mixed studies reviews. This tool reflects best practices in mixed methods research at the time of the tool's development. This tool is well suited to a public health context, particularly for questions related to complex interventions that are context-dependant and process-oriented. Systematic mixed studies reviews leverage the strengths of qualitative and quantitative evidence, which allows for a comprehensive understanding of public health issues and solutions.

Description

The Mixed Methods Appraisal Tool (MMAT) is a checklist that was developed to provide a quality appraisal tool for quantitative, qualitative and mixed methods studies included in systematic mixed studies reviews (Pluye & Hong, 2014). Compared to other tools, the MMAT specifically includes criteria for appraising mixed methods studies. While critical appraisal tools are more widely available for quantitative and qualitative research, there has not been consensus on quality criteria for mixed methods research (O'Cathain et al., 2008). Mixed methods research is important because it provides both breadth and depth of evidence for specific questions of interest. Quantitative methods provide statistical evidence to draw generalizable descriptions and causal inferences, whereas qualitative methods provide foundational or additional contextual, process and experiential evidence.

Whenever possible, good quality syntheses of the evidence, rather than single studies, should be used for public health practice and policy decisions with regard to prevention, diagnosis and treatment of health problems, and health promotion. Mixed studies reviews synthesize quantitative, qualitative and mixed methods studies that relate to a particular question. As with other types of syntheses, reviews of mixed studies save time and effort because the different quantitative, qualitative and mixed methods studies have already been found and the findings summarized for specific questions of interest (Ciliska, 2012).

The Mixed Methods Appraisal Tool (MMAT) provides a set of criteria for concomitantly appraising the methodological quality of quantitative, qualitative and mixed methods studies included in a mixed studies review. The criteria for the initial tool was based on

(a) the theoretical work of Ian Hacking, a philosopher of sciences and leader of the social constructionism movement and

(b) a review of 17 health-related systematic mixed studies reviews (Pluye et al., 2009).

The criteria for this 2011 version of the MMAT were further informed by methodological experts and workshops with national and international researchers. The MMAT checklist includes screening questions which are applied across all relevant studies. There are 19 items to assess the quality of five different types of studies (qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies). An overall quality score can be ascertained using this tool for each included study. Given that such score may not be informative, users can also create a descriptive summary using MMAT criteria.

Implementing the Method/Tool
Steps for Using Method/Tool
  1. Apply the screening questions for all studies.
  2. For each relevant study, determine the type of design, and use the corresponding criteria to appraise a study's quality.
  3. Two independent reviewers should conduct the appraisal process. Reviewers should meet before the appraisal process to share their understanding of the MMAT checklist and manual to increase the rigor of the quality appraisals.
  4. To determine an overall quality score for each study (if needed), divide the number of criteria met by the total number of criteria as per the appropriate methodological quality criteria for quantitative, qualitative and mixed methods studies.
Who is involved

Systematic review authors who are interested in appraising and reporting on the quality of studies included in their mixed studies reviews could use this tool. The tool may also be useful for knowledge users who would like to appraise the quality of primary studies with diverse designs (qualitative, quantitative and mixed methods research designs).

Conditions for Use
Not specified
Evaluation and Measurement Characteristics
Evaluation
Has been evaluated.

While the MMAT is still currently in development and therefore should be used with caution, it has gone through a number of iterative evaluations since the development of the initial MMAT version in 2009.

Validity
Validity properties meet accepted standards.

One evaluation of the MMAT's validity has been published to date (Pluye, 2009). This evaluation used a theoretical model and qualitative thematic data analysis of the quality appraisal procedures used in 17 mixed systematic reviews in health sciences. Findings from this evaluation support the substantive/theoretical and content validity of the initial criteria.

Reliability
Reliability properties meet accepted standards.

The kappa statistic measures the level of agreement between two observers that could be expected beyond chance. Using the Landis and Koch (1977) criteria, Kappa scores > 0.8 are considered to be almost perfect agreement (Kappa between 0 and 0.20 indicates slight agreement; between 0.21 and 0.40 indicates fair agreement; between 0.41 and 0.60 indicates moderate agreement; between 0.61 and 0.80 indicates substantial agreement). Two evaluations of the MMAT's reliability have been completed to date (Pace et al., 2012; Souto et al., 2014). Results from both evaluations showed that agreement between reviewers was fair to perfect on MMAT criteria. Results from Pace et al. (2012) showed that the overall quality score of appraised studies was substantial.

Methodological Rating
Moderate
Method/Tool Development
Developer(s)

Pierre Pluye

Method of Development

A three-step critical review was conducted.

Release Date
2011
Contact Person/Source

Pierre Pluye

Department of Family Medicine, McGill University

Email: pierre.pluye@mcgill.ca

Resources

Title of Primary Resource
Proposal: A mixed methods appraisal tool for systematic mixed studies
File Attachment
None
Web-link
Reference
Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., Boardman, F., Gagnon, M.P., & Rousseau, M.C. (2011). Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Retrieved from http://mixedmethodsappraisaltoolpublic.pbworks.com. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ
Type of Material
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Copyright Clearance Center

Title of Supplementary Resource
The quality of mixed methods studies in health services research
File Attachment
None
Web-link
Reference
O'Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy, 13(2), 92-98.
Type of Material
Journal article
Format
Periodical
Cost to Access
Language
English
Conditions for Use
Copyright © 2014 by SAGE Publications

Title of Supplementary Resource
The measurement of observer agreement for categorical data.
File Attachment
None
Web-link
Reference
Landis, J,R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
Type of Material
Journal article
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

Title of Supplementary Resource
Systematic Mixed Studies Reviews: Reliability Testing of the Mixed Methods Appraisal Tool.
File Attachment
None
Web-link
Reference
Souto, Q.R., Khanassov, V., Pluye, P., Hong, Q.N., Bush, P., & Vedel, I. (June 28, 2014). Systematic Mixed Studies Reviews: Reliability Testing of the Mixed Methods Appraisal Tool. Mixed Methods International Research Association Conference (MMIRA), Boston, USA.
Type of Material
Conference presentation
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

Title of Supplementary Resource
Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review.
File Attachment
None
Web-link
Reference
Pace, R., Pluye, P., Bartlett, G., Macaulay, A.C., Salsberg, J., Jagosh. J., et al. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. International Journal of Nursing Studies, 49(1):47-53.
Type of Material
Journal article
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Copyright © 2011 Elsevier Ltd. All rights reserved.

Title of Supplementary Resource
A Compendium of Critical Appraisal Tools for Public Health Practice (Revised)
File Attachment
None
Web-link
Reference
Ciliska, D., Thomas, H., & Buffet, C. (2012). A Compendium of Critical Appraisal Tools for Public Health Practice (Revised). [tool]. Hamilton, ON: National Collaborating Centre for Methods and Tools. [http://www.nccmt.ca/pubs/CompendiumToolENG.pdf]
Type of Material
Online Resource
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use

Title of Supplementary Resource
The quality of mixed methods studies in health services research
File Attachment
None
Web-link
Reference
O'Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy, 13(2): 92-98.
Type of Material
Journal Article
Format
Periodical
Cost to Access
Language
English
Conditions for Use
Copyright © 2014 by SAGE Publications

Title of Supplementary Resource
Welcome to the public wiki 'Mixed Methods Appraisal Tool'
File Attachment
None
Web-link
Reference
Welcome to the public wiki 'Mixed Methods Appraisal Tool'. (2014). In PBworks. Retrieved July 25, 2014, from http://mixedmethodsappraisaltoolpublic.pbworks.com/w/page/24607821/FrontPage
Type of Material
Website
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Not specified

Title of Supplementary Resource
Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews
File Attachment
None
Web-link
Reference
Pluye, P. & Hong, Q.N. (2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35: 29-45.
Type of Material
Journal Article
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Copyright Clearance Center

Have you used this resource? Share your story!