Research comes in many forms. Similarly to how choosing the right wine to pair with a meal can enhance the experience, choosing the right study type for your research can enhance how the research question is answered.
Different study designs exist to serve different purposes. Depending on the question you are asking, a study design looking into the past or projecting into the future may be most suitable. Potentially a case study, providing a well-rounded, holistic picture may best represent the effect of an intervention. Additionally, the method and type of data collected also has to be suitable to be able to draw conclusions with confidence.
As a chiropractic research organisation, we have a vested interest in both the quality and the quantity of chiropractic research. Now, it is without doubt that recent decades have seen advancements in research techniques and processes across the board, and chiropractic is no exception. However, when a paper emerges looking at the risk of biases in chiropractic mixed methods research, it’s well worth sitting up and taking notice.
First of all: What is mixed methods research?
Mixed methods research is a research approach that combines both quantitative and qualitative approaches in a single study. The use of this design is not particularly new, with evidence of its use in the 1800s, it is gaining popularity. [1] Including multiple methods of data collection and analysis can broaden and deepen our understanding of findings. It is therefore no surprise that there has been an uptick in the number of mixed methods studies published. It still remains a relatively uncommon approach, compared to the volume of other designs and approaches published. These approaches are well-suited to assessing multilevel programs and interventions, including chiropractic research problems.
Every research design and approach has recommended guidelines that ensure the validity and reliability of the findings obtained from the study. As you can imagine, increasing the number of methods utilised within a study not only adds value but also increases the complexity. Often mixed methods research requires additional time and resources, ideally including an expert in qualitative and quantitative measures or a mixed methodologist as a part of the research team. As a mixed method approach has gained popularity recently, many journals are yet to adopt relevant appraisal guidelines, and as such many authors have not been required to comply with published methodological standards. This growing trend of a multi-method approach has also been observed within chiropractic research.
The study
A systematic review was recently published in the Journal of the Canadian Chiropractic Association and addressed the concern about suboptimal quality of mixed methods chiropractic research. The review screened and identified 1040 citations, 65 of which met the eligibility criteria for review. 10 studies were found to report qualitative and quantitative results in separate articles, and as such 55 unique mixed methods studies were analysed for their risk of bias. The review assessed the articles against the Mixed Methods Appraisal Tool (MMAT). The MMAT has been validated as a quality appraisal tool for systematic reviews, and consists of 11 appraisal items.
Characteristics of the studies:
- 80% of the studies were conducted in, or had corresponding authors from, the US, Canada, and Australia.
- 53% of the studies had 4 or fewer authors
- 75% of the studies were published after 2010
- 53% of the studies were published in journals that had an impact factor
- 45% of the studies included a methodologist among their authors.
What was found?
Only 2 of the 55 studies met all 11 MMAT criteria for methodological quality in mixed methods research. 11% of studies met 10 criteria, 20% met at least 8, and most studies met 7 or less criteria. When the reviews considered which factors may be associated with an increased risk of bias they found the following:
- Studies published since 2010, studies published in journals with impact factors, those that included more than 4 authors, and those that included a methodologist among the study team were associated with a lower risk of bias.
- When considering multiple variables at once, it was found that studies published since 2010 and those published in journals with an impact factor remained associated with a lower risk of bias.
An average of 60% of the quality criteria in mixed methods research were addressed across the 55 studies reviewed. The impact of the research setting or of the researchers themselves on the qualitative outcomes, as well as the limitations of combining qualitative and quantitative methods were poorly considered in approximately 75% of articles. Additionally, 40% of studies also failed to provide sufficient detail about allocation concealment, instrument validation, or assessment of selection bias, or describe the integration of the different methods. Follow up rate in one third of the studies were inadequate.
What does this mean for chiropractic research?
These findings are consistent with the results of reviews in other healthcare fields, including complementary and alternative medicine and nursing. The methodological strength in many of the mixed methods studies included in these reviews was found to be unsatisfactory across various MMAT items.
Previous reviews have found a statistically significant association between a lower risk of bias and a higher number of authors or the inclusion of methodologists. The authors identify the scope of the current review was broadened to include additional areas of methodological expertise, such as expertise in qualitative research, epidemiology, or statistics as most studies did not clearly disclose the inclusion of a mixed methodologist. Potentially, if methodologists were included in the research team and disclosed, a more accurate association can be identified.
There is clearly a need for stronger mixed methods research in the chiropractic, and broader healthcare, fields. Compliance with critical appraisal guidelines have been associated with reduced methodological bias in research studies, which is further enhanced when the authors are required to meet these standards as a condition of submission. The authors of the review suggest editorial review boards of journals have an important role to play in improving the quality of conduct in mixed methods studies. Inclusion of a mixed methodologist on their editorial board and incorporating mixed methods appraisal tools into the peer review process are two ways actionable.
While the results of the review were consistent with other healthcare fields, and certainly not more damning to one field than another, it is well worth bearing in mind as chiropractic research advances.
REFERENCES:
- Mertens, D. Mixed methods in evaluation: history and progress. In Mixed methods design in evaluation. SAGE Publications. 2018. Chapter 1;1-30
- Emary PC, Stuber KJ, Mbuagbaw L, Oremus M, Nolet PS, Nash JV, et al. Risk of bias in chiropractic mixed methods research: a secondary analysis of a meta-epidemiological review. The Journal of the Canadian Chiropractic Association. 2022. 66(1):7–20.
