| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
PERSPECTIVE |
1 From the Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada (DM and ACT) and the Department of Epidemiology & Community Medicine, Faculty of Medicine (DM), and the Institute of Population Health, University of Ottawa, Ottawa, Canada (ACT)
2 Supported by the International Life Sciences Institute Research Foundation, a University of Ottawa Research Chair (to DM), and a Canadian Institutes of Health Research Canada Graduate Scholarship and a University of Ottawa Excellence Scholarship (to ACT). The sponsor had no involvement in the writing of the manuscript or the decision to submit the manuscript for publication. 3 Reprints not available. Address correspondence to D Moher, Clinical Epidemiology Program, Ottawa Health Research Institute, Box 208, 501 Smyth Road, Ottawa, ON K1H 8L6, Canada. E-mail: dmoher{at}uottawa.ca.
ABSTRACT
Systematic reviews (SRs) are an increasingly popular evidence-based tool and are often used to answer complex research questions across many different research domains. Early SR methodology was advanced by social scientists, and the term meta-analysis was coined by a social scientist who also conducted research in psychology. SRs have recently become popular in healthcare and are likely to be beneficial in any field. The aim of this report is to highlight issues in SR conduct with a focus on the field of nutrition and to make recommendations on improving SR conduct in this area. Development of the research question is probably the most important step in conducting an SR. The 4 main components of an answerable question are 1) the patient, population, or problem; 2) the intervention, independent variable, or exposure; 3) the comparators; and 4) the dependent variables or outcomes of interest. The question will be used to determine the optimal methods for conducting the SR. SRs often include study designs beyond randomized trials and do not always include a meta-analysis of the results. Other topics explored include understanding and interpreting discordant reviews and the importance of reporting tools [eg, QUality Of Reporting Of Meta-analyses (QUOROM Statement) or CONsolidated Standards Of Reporting of Trials (CONSORT Statement)]. Recommendations are then provided, such as developing a capacity-building program, searching the primary literature for research gaps, and extending reporting tools such as the QUOROM Statement to the field of nutrition.
INTRODUCTION
Synthesizing evidence from research dates back to the early 1900s, yet improved methods for evidence synthesis have been developed only within the past few years (1). Early systematic review (SR) methods were advanced by social scientists (1), and the term "meta-analysis" was coined in 1976 by a social scientist who also did research in the area of psychology (2). Although the importance of evidence synthesis in medicine was recognized in the 1970s (3), widespread use of these techniques did not occur until 2 decades later (1). Potentially contributing to this "movement" was evidence that the judgments and opinions of experts were often biased. A pivotal study by Antman et al (4) concluded that effective therapies were not mentioned, whereas ineffective and potentially harmful therapies often were recommended by experts. The term "evidence-based medicine" was coined only 
15 y ago (5).
Over the past 2 decades, SRs have become increasingly popular in evidence-based healthcare (6). Policy makers, clinicians, and others have little time to keep up with the published literature. SRs offer a convenient way for them to keep up-to-date with the current evidence. Furthermore, SRs are often used in the development of clinical practice guidelines (7) and are increasingly viewed as a useful tool for health decision makers (8, 9). The increased utility of SRs has probably contributed to an increase in their publication rates. It was recently estimated that, each year, MEDLINE cites 2500 new SRs in English (10)—a rate 166 times that reported in 1991 (11).
Because of the utility of SRs, the SR approach is likely to benefit any field (12), including nutrition. The aims of this report are to highlight issues in SR conduct with a focus on the field of nutrition and to make recommendations for improving SR conduct in this area.
METHODS
Terminology
The terminology used to describe an SR and a meta-analysis has evolved over time. There is no consistent use of these terms, but we will adopt the definitions used by The Cochrane Collaboration (13). An SR consists of a clearly formulated question and explicit methods to identify, select, and critically appraise relevant research and collects and analyzes data from the studies that are included in the review. A meta-analysis is the use of statistical techniques in an SR, which integrates the results of included studies. Thus an SR does not necessarily include a meta-analysis.
Other types of reviews, such as narrative reviews, will not use the explicit methods outlined in the definition proposed by The Cochrane Collaboration (Table 1
; 10). As such, these other types of reviews may be susceptible to bias (4). A properly conducted SR is much more resource- and labor-intensive than is a narrative review (14).
|
Developing a systematic review question
SRs are currently being conducted in many different areas and have been used to address many types of research questions (Table 2). For example, an SR was used to answer the following question within the crime and justice domain: "Does a neighborhood watch reduce crime?" (15). Another SR was used to answer the question within education: "Which approaches to parental involvement improve the academic performance of elementary school–age children?" (16). SRs have been used to answer many complex questions within the field of nutrition, including "Which community-based nutrition and physical activity interventions are effective at preventing chronic disease in low-income populations?" (17); "What are the long-term effects of advice to restrict dietary sodium in adults with and without hypertension?" (18); and "What is the association between a diet rich in whole grain, bran, and germ and the risk of type 2 diabetes?" (19). Other real-life examples of questions answered by SRs are provided elsewhere (12).
|
An example of the use of these components in creating a question is that, to examine the sodium intake in the United States, the question could be phrased as "What are the health-related effects (O) associated with a high-salt diet (I) compared with a low-salt diet (C) among persons living in the United States (P)?" To examine the association between fruit and vegetable intake and cardiovascular disease among diabetics, the question could be phrased as "Among persons with diabetes (P), what is the association of a high fruit and vegetable diet (I) compared with a low fruit and vegetable diet (C) with respect to myocardial infarction, stroke, and cardiovascular disease–related mortality (O)?"
The explicit methods involved in conducting an SR are dependent on the question that is being asked. Literature can be searched according to the PICOS components. The eligibility criteria (ie, inclusion and exclusion criteria) will be based on the question. The data abstraction instrument or instruments will be developed to abstract from the included studies the data that will be used to answer the question. The synthesis of results abstracted from the included studies will also help provide an answer to the question. The methods used in an SR may vary depending on the type of question and the requirements of the organization conducting or commissioning the review. Three examples of such an organization include the Cochrane Collaboration, the Campbell Collaboration, and the Evidence-based Practice Center (EPC) program.
The Cochrane Collaboration (13) is a global entity including 10 000 members organized into Clinical Review Groups (eg, Schizophrenia Group), Methods Groups (eg, Bias Methods Group), and Fields (eg, Diet and Nutrition Subfield of the Cochrane Primary Health Care Field). The mission of The Cochrane Collaboration is to conduct SRs in all areas of healthcare, including nutrition. Cochrane reviews are completed by using an approach described in the Cochrane Handbook (13). A distinctive characteristic of Cochrane reviews is that they often answer questions regarding the effectiveness or efficacy of an intervention (ie, "what works"). As such, The Cochrane Collaboration relies strongly on the synthesizing of evidence from randomized controlled trials (RCTs). However, there are some exceptions to this within Cochrane. An example is the Effective Practice and Organisation of Care (EPOC) group, which often addresses broader questions and includes other study designs.
RCTs are given high prominence within the Cochrane Collaboration because they minimize the influence of bias in their results. Whereas there is considerable evidence to support this perspective and the importance of randomized trials, there also are data indicating that such study designs account for only 10% of the healthcare literature (28, 29). In contrast, the Campbell Collaboration, the sister group to The Cochrane Collaboration, addresses broad questions related to crime and justice, education, and social welfare and includes all types of study designs (eg, RCTs, cross-sectional studies, case-control studies, cohort studies, and case series) (Table 2
). Campbell reviews are conducted according to the guidance provided on the organization's website (Internet: www.campbellcollaboration.org/guidelines.asp).
The EPC program (30), started in 1997, is funded by the Agency for Healthcare Research and Quality and consists of 14 centers throughout North America (Internet: http://www.ahrq.gov/clinic/epcix.htm). The remit of this program is to complete SRs in a wide variety of healthcare and policy areas, and guides for conducting these reviews have been proposed (32). As do those of the Campbell Collaboration, these SRs address broader types of questions and thus are not limited to reports of RCTs. A more detailed discussion of the EPC program can be found elsewhere (31).
Whereas The Cochrane Collaboration, Campbell Collaboration, and EPC program are high-profile, 
100 other groups conduct reviews, commission SR conduct, or do both (33). Examples include the Canadian Agency for Drugs and Technologies in Health (Internet: cadth.ca/index.php/en/hta/programs/health-technology-assessment/process), the Centre for Reviews and Dissemination (York University, York, United Kingdom; Internet: www.york.ac.uk/inst/crd/report4.htm), and, specific to the field of nutrition, the American Dietetic Association (Internet: www.eatright.org/cps/rde/xchg/ada/hs.xsl/education_13146_ENU_HTML.htm). All of these organizations conduct SRs by using different templates.
FINDINGS
Differences in questions or conduct may lead to differences in results and conclusions
Variations of the SR question or differences in the SR methods used may explain why SRs addressing a similar topic have different, sometimes startlingly different, results or conclusions, or both; these are called discordant reviews. An example of discordant reviews is apparent among 4 SRs examining the cardiovascular effects of vitamin E supplements (Table 3; 34-37). Although the SR questions were similar, variations are apparent. For example, one of the reviews (35) focused on effectiveness, whereas another (36) focused on efficacy. Furthermore, different methods were used to conduct the reviews. For example, one SR (36) searched multiple databases, whereas another (34) searched only one database. Different inclusion or exclusion criteria, language limitations (eg, including only reports in English), and outcomes of interest also were evident. The number of included studies varied from 7 (34, 35) to 84 (36).
|
). However, the fourth review (37) conducted a dose-response analysis, for which high doses of vitamin E (500 IU/d) were shown to significantly increase the risk of all-cause mortality by 9–14%. Two of the SRs (34, 36) concluded that vitamin E had no benefit with respect to cardiovascular events, and one SR (35) concluded that vitamin E had neither benefit nor harm with respect to cardiovascular events. However, the fourth review (37) concluded that a dose-response relation between vitamin E and all-cause mortality was observed (ie, that vitamin E at high doses is harmful).
For an understanding of discordant reviews, an assessment of the risk of bias (ie, study quality) of the SRs is likely to be beneficial (38). Such an exercise was completed for the 4 SRs discussed above by using the Oxman and Guyatt (39) instrument (Table 4). This validated instrument consists of 9 main criteria for assessing the scientific quality of review articles. The final item asks the assessor to rate the overall scientific quality of the SR by using a score ranging from 1 (ie, extensive flaws) to 7 (ie, minimal flaws). When the Oxman-Guyatt tool was applied to these 4 SRs, the scores ranged from 1 [indicating extensive flaws in the SR (35)] to 7 [indicating minimal flaws in the SR (36)]; the other 2 SRs fell between the extremes.
|
As it would be for any type of research, reporting SRs to the highest possible standard is critical, because the report is one of the most important ways of disseminating the results to the broader research community and beyond. Dissemination is likely to be more widespread if the report accurately and transparently describes the research methods and results. It is important that the report of a research endeavor is, in many instances, the only public record of the existence of the research itself. High-quality reports enable readers to critically interpret and use the results. There also are emerging data indicating that the use of reporting guidelines improves the quality of the research reports (40).
To help improve the quality of reporting of all research, the UK National Health Service National Library for Health and the UK National Institute for Health Research recently provided funds to set up the EQUATOR Network (Internet: www.equator-network.org). The objective of this network is to improve the quality of scientific publications by promoting transparent and accurate reporting of health research and by supporting the development of high-quality reporting guidelines. An early project completed by the network executive was an SR that identified existing reporting guidelines and a survey that identified issues faced by those developing such guidelines (41). More than 50 sets of reporting guidelines were identified; they can be accessed via the EQUATOR Network website.
Many studies have evaluated the quality of SR reports. In 1987, Mulrow (42) examined 50 review articles published in 1985 and 1986 in 4 leading medical journals; he found that none met all 8 explicit scientific criteria that they examined, such as a quality assessment of included studies. Also in 1987, Sacks et al (43) evaluated the adequacy of reporting on 23 characteristics in 6 domains in 83 meta-analyses. They found that reporting generally was poor; between 1 and 14 characteristics (
± SD: 7.7 ± 2.7) were adequately reported. A 1996 update of this study found little improvement (44). To address the suboptimal reporting of meta-analyses, an international group developed guidance on the quality of reporting of meta-analyses of RCTs, called the QUOROM Statement.
The QUOROM Statement consists of a 21-item checklist that documents the process of completing a meta-analysis and a flow diagram that details the number and status of included articles at each stage of the meta-analysis process (45). Since the QUOROM Statement was published in 1999, the evidence base underlying the conduct of SRs has matured. In addition, a more comprehensive understanding of some conceptual issues, advances in methodology, practical innovations in the conduct and reporting of SRs, and changes in terminology have occurred. It is for these reasons that the QUOROM Statement was updated in June 2005. It is now called the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement, and it applies not only to SRs of RCTs but also to any SR assessing the effectiveness of any intervention.
Quality of reporting of systematic reviews within nutrition
We completed an extensive search to identify studies that have examined the general quality of reporting of SRs within nutrition; we found that few studies exist, perhaps because there are few individual studies (eg, cohort studies or RCTs) to be synthesized. More likely the reasons are multifactorial, such as a lack of capacity to conduct SRs on nutrition topics and possibly a lack of clarity as to how best to conduct and report such reviews.
There are limited data on the specific issues that systematic reviewers in the nutrition domain should specifically address. Gibson et al (46) noted that the following steps were involved in nutrition assessment during the conduct of RCTs and meta-analyses: measuring food intakes via qualitative or quantitative means (eg, food diary or food recalls); converting food intakes to observed intakes of nutrients by using food-composition tables; and evaluating nutrient adequacy. Challenges encountered during a series of reviews of omega-3 fatty acids were also documented (47), including difficulties in sorting through the numerous endpoints reported in the included studies and variations in the source of dietary intake (eg, supplement or dietary component), study design, and study duration of the included studies. These challenges highlight how important it is for those completing individual studies to ensure that these points are adequately integrated into the conduct of such studies.
A systematic review is dependent on the quality of included studies
The successful conduct of any SR is dependent to a large degree on the quality of the included studies. Historians may well view the first 50 y of reporting of RCTs with some surprise. They will encounter what might be described as a cognitive dissonance: a disconnect between the increasing sophistication of the design and the swelling cost of these studies and the apparent lack of care—a disastrous lack in some cases—with which they have been reported. A series of studies beginning in 1995 found empirical evidence that results may be biased when trials use inferior methods or are reported without adequate description of the methods; notably, failure to conceal the allocation process is associated with a 30% exaggeration in the effectiveness of an intervention (48). The cause for concern is obvious: if the conduct or reporting of RCTs is poor, treatments may be introduced that are less effective than was thought or that may even be ineffective, and these reports are included in SRs.
This concern led to the Consolidated Standards of Reporting of Trials (CONSORT) Statement recommendations originally published in 1996 and updated in 2001; anther update is currently in progress (Internet: www.consort-statement.org). The CONSORT Statement consists of a 22-item evidence-based checklist that authors can use to guide the reporting of their trials and a flow diagram to account for the flow of participants throughout the trial process. The CONSORT Statement has been extended in several directions, such as cluster trials, equivalence and noninferiority trials, and nonpharmacologic treatments. These extensions have been made because of inadequate reporting of important study aspects not covered by the original CONSORT Statement. The CONSORT statement for herbal interventions was developed to help improve the quality of reporting of such studies, which may also have implications for how those studies, as well as observational ones, are designed and conducted. Currently, the CONSORT Statement has not been widely implemented in the field of nutrition.
SUMMARY AND RECOMMENDATIONS
SR techniques have been used to answer complex research questions across many different research domains. SRs have recently become popular within healthcare and are likely to be beneficial in any field, including nutrition. Developing the research question is probably the most important step in conducting an SR. SRs often include study designs beyond randomized trials and do not always include a meta-analysis of the results. As with any other type of study, the optimal conduct and reporting of SRs is necessary.
To strengthen the scientific rigor of nutrition science research endeavors, we propose the following ideal recommendations, recognizing that their adoption may be difficult without the involvement of an expert panel akin to the Institute of Medicine (Table 5).
|
Second, to ascertain which nutrition topics are most likely to benefit from synthesis (ie, an SR), we recommend that a comprehensive search of the primary literature (eg, cohort studies) be conducted and that the resulting reports be categorized according to the nutrition-related topic. When there is a sufficient evidence base, we recommend that this priority list form the basis of SRs that should be commissioned for immediate completion. Such an exercise is also likely to provide evidence for locating the gaps in the evidence base and thus to give a rationale for the areas in which primary research could be earmarked for investment. To identify gaps in the SR literature and generate new SR questions, we also recommend the development of a repository of SRs within the field of nutrition.
Third, SRs are most likely to be useful if the quality of reporting of the individual studies included is sufficiently high; this is a longer-term undertaking than some of the other recommendations. We recommend that the CONSORT Statement be implemented for nutrition-related clinical trials (Internet: www.consort-statement.org). Such an initiative for herbal interventions could build on the CONSORT Statement. Similarly, where appropriate, we recommend that other reporting guides be extended or implemented for this field (Internet: www.equator-network.org). We also recommend that journals publishing nutrition-related research endorse evidence-based reporting guidelines to improve the quality of nutrition research in their journals.
Fourth, we recommend that the PRISMA Statement be extended to the reporting of SRs that evaluate nutrition (45). We also recommend that journals publishing SRs on nutrition-related topics endorse the PRISMA Statement to improve the quality of reporting of this research in their journals.
Fifth, because the conduct and reporting of SRs are so intertwined, we recommend that an evidence-based approach to developing a standard guide for conducting SRs within the nutritional science domain be undertaken. Preexisting SR conduct guides providing some insight into the nutrition field, such as the Cochrane Handbook (13), can be used as a starting point in the development of this standard guide. It would also be beneficial to include persons with expertise in many different areas to develop evidence-based guidelines for the best reporting and conduct of nutrition-related SRs. These persons may include a broad base of funders and those who commission nutrition-related SRs, those actively involved in obtaining empirical evidence to guide the conduct of nutrition-related SRs, journal editors who publish SRs, and persons with experience and expertise in conducting SRs in the nutrition field. Such a consortium is likely to have a monumental effect in improving the reporting and conduct of nutrition-related SRs.
Sixth, we recommend that an annual review of the quality of nutrition literature be conducted. Such a review will provide important stakeholder groups with an evidence base as to the current quality of this important literature.
ACKNOWLEDGMENTS
We thank the attendees of the Evidence-based Approaches to Developing Dietary Guidance Workshop, hosted by the International Life Sciences Institute Research Foundation in December 2007, for their useful feedback on a preliminary version of the manuscript. We also thank members of Chalmers Research Group for their feedback on Table 1.
The authors' responsibilities were as follows—DM: conceptualized the study and wrote the manuscript; ACT: helped write the manuscript and edited it. Neither author had a personal or financial conflict of interest.
REFERENCES
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
