What was the goal of this work?
The original purpose of this work was to test the hypothesis that individuals with multiple chronic conditions are underrepresented in RCTs of behavioral and psychosocial interventions published in general medical, behavioral medicine, behavioral science, health psychology, social science, and public health journals.
This review had the following initial goals:
Goal 1: Conduct a systematic review to assess the frequency with which research participants with MCC are represented in all or a representative subset of RCTs of behavioral and psychosocial interventions published in general medical and specialized journals, published within the last decade or decade and a half, that focus on behavioral medicine and behavioral science, health psychology, social science, and public health.
Goal 2: Determine whether there are significant differences by type of journal or over time in the frequency with which research participants with MCC are represented in RCTs of behavioral and psychosocial interventions.
Are there any restrictions on the use of the data/content of this website?
We highly encourage researchers to use the content of this website! However, we do ask that you include proper citation (see “How do I cite the online database?”) Also, when downloading the database file, we require the first and last name, email address and institutional affiliation of the person downloading. This information is used to track interest in our data. There is a possibility that we will contact you in the future to follow up on your usage of the data.
Who funded this project?
This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under Contract No. HHSN261200800001E. The content of this publication does not necessarily reflect the views or polices of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations to imply endorsement by the U.S. Government.
What is a “Systematic Review”?
A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made. Key characteristics of a systematic review are:
1. a clearly stated set of objectives with pre-defined eligibility criteria for studies
2. an explicit, reproducible methodology
3. a systematic search that attempts to identify all studies that would meet the eligibility criteria
4. an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias
5. a systematic presentation, and synthesis, of the characteristics and findings of the included studies.
Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
Why isn’t our study included in your database?
The 600 studies included in our database are a representative sample of the studies from 2000-2014 that meet our selection criteria. Since these criteria encompass a broad range of studies, it was not feasible to include each one.
Three separate literature searches were performed within defined time periods (2000-2004, 2005-2009, 2010-2014). Within each time-period group, search results were randomly ordered using the RAND function in Microsoft Excel (2013). The study selection process was performed on the randomly ordered results until the desired number of studies for extraction meeting inclusion/exclusion criteria was identified (200 per time period for a total of 600 articles). Selection was stratified by time period to ensure that an adequate number of studies per time period were selected to allow for analysis over time.
How should I cite this work?
Please use the following citation:
Stoll C, Izadi S, Colditz G, Systematic Reviews to Inform Research and Treatment for Multi-Morbidities. Prepared by Washington University School of Medicine in St. Louis under Subagreement 14X262 of Contract No. HHSN261200800001E. Frederick, MD: Leidos Biomedical Research, Inc; October 2015.
How was the data organized?
Study data were collected and managed using REDCap electronic data capture tools hosted in the Biostatistics Division of Washington University School of Medicine. REDCap (Research Electronic Data Capture) is a secure, web-based application designed to support data capture for research studies, providing: 1) an intuitive interface for validated data entry; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for importing data from external sources.
REDCap was chosen because it is uniquely suited to meet the needs of an effective and efficient data extraction process. It allows form creators to require specific entry formats for individual questions (ensuring reviewers input data in a consistent format), allows for multiple types of response formats (dropdown menus, select all, select only one, open-ended text entry, etc), and performs data validation to improve accuracy of extraction. In addition, REDCap has a built in “Data Comparison Tool” feature, which allows for the differences in two records to be identified automatically.
How were readers trained?
Readers were Master of Public Health students or had equivalent experience. Their previous experience included managing behavioral research projects, delivering behavioral and/or psychosocial interventions, and clinical experience treating chronic conditions. Readers went through an extensive training process before they began extracting data. Training focused on relevant background material and important concepts such as background information on MCC, systematic review methods, RCT methods, RCT reporting, and assessment of bias in RCTs, as well as the specific extraction items and process for this review. Readers were assessed for accuracy and completeness of extraction on multiple test articles before beginning to extract included articles so that any learning curve did not affect the quality of the data.
Training materials can be found at http://www.mccsystematicreview.wustl.edu/Other-Materials
What was the process of data extraction?
Each article was extracted independently by two reviewers. This method was chosen as systematic review best practices recommend that independent data extraction by more than one reader can minimize errors and reduce potential biases being introduced by readers. After both extractions were complete, any differences were identified using the REDCap Data Comparison Tool. Disagreements were resolved by a third party with graduate training in public health and four years of experience and expertise in extraction of data from published reports, the design and implementation of systematic reviews, and the goals and objectives of this review. This continuous comparison reduced error in the database, and allowed the monitoring of any systematic errors by readers, which helped to ensure the high quality of the extraction process and of the data.
Readers were assigned articles to extract and the order of extraction was designed to ensure that number of articles extracted was kept constant within time periods to minimize any bias created by extraction over a long period of time. Additionally, as two readers were required per article for double extraction, readers were shuffled in regards to which additional reader they were matched with in order to better identify inconsistencies across readers.
How were variables selected for extraction?
The variables extracted were designed to assess inclusion and reporting of MCC in all phases of a trial– eligibility criteria, participant screening, participant selection and reporting of characteristics, and study analysis. A thorough review of relevant peer-reviewed literature, including reviews of a similar nature or those that assessed the inclusion criteria of RCTs, was performed to identify key variables previously used to evaluate inclusion of specific populations in RCTs.
Briefly, the items extracted consisted of basic study characteristics (author, title, journal, journal type, year, region, study registration, sample size), intervention details, eligibility information, participant selection details, study outcomes, and risk of bias assessment.
How was study quality assessed?
The risk of bias of each included study was assessed for the purposes of describing and evaluating the body of evidence that conclusions are based on, and assessing if study quality and inclusion of MCC are associated. This risk of bias was assessed using a modified version of the Cochrane Collaboration’s Risk of Bias tool. The Risk of Bias tool includes six domains: selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias.21 For each item the reader categorizes the article as low risk of bias, high risk of bias, or unclear risk of bias. The unclear risk of bias option allows the reader to indicate when the article has not provided enough information to make a judgement, which helps separate out issues of study reporting quality from actual study quality.
The Cochrane Collaboration’s Risk of Bias tool was adapted slightly for use in this review. Specifically, the “other bias” domain was removed as this element was unnecessary under the scope of this review. The inclusion of this domain is beneficial when considering more specialized studies, therefore it was not necessary for our generalized approach. For attrition bias, readers made judgements on whether attrition was appropriately reported and explained but they did not assess the actual amount of attrition. As this review encompassed studies on a wide variety of populations, interventions, and outcomes, it was not feasible for readers to have enough domain specific knowledge to adequately assess how the amount of attrition in each study may have affected the outcome.
How was a representative subset of RCTs of behavioral and psychosocial interventions published in general medical and specialized journals, published from 2000-2014, that focus on behavioral medicine and behavioral science, health psychology, social science, and public health created?
In order to create the desired representative subset we first employed a thorough search strategy in order to identify all relevant published RCTs. The librarian designed search strategies to retrieve, as thoroughly as possible, all reported randomized controlled trials in adults regarding chronic illness in PubMed Medline and Embase. Three separate literature searches (using identical keywords and in the same databases) were performed within the defined time periods (2000-2004, 2005-2009, 2010-2014). This was done in order to ensure that we would be able to consider change over time. Within each time period, search results were randomly ordered using the RAND function in Microsoft Excel (2013). The study selection process (application of eligibility criteria onto each article) was performed on the randomly ordered results until studies meeting selection criteria were identified for extraction (200 per time period for a total of 600 articles).
What were the eligibility criteria for this review?
The following eligibility criteria was used for this review:
1. RCT with original data
2. Primary report (not protocol, posttrial follow-up study, secondary/subgroup analyses, etc.)
3. Published in English
4. Targeting at least one of the chronic conditions of interest (i.e. all enrolled participants must have at least one of the 20 specified chronic conditions)
5. Had a primary goal to develop and/or test efficacy or effectiveness of a behavioral or psychosocial intervention
6. Enrolled participants and applied eligibility criteria at the individual level
7. Enrolled only adult subjects (18+)
Eligibility criteria were designed to be fairly broad to produce a heterogeneous sample of the literature in terms of populations, interventions, and outcomes.
How was “behavioral or psychosocial intervention” defined for the purpose of this review?
After considering the goals of the review, we elected to define “behavioral or psychosocial intervention” as any intervention that is non-pharmacological and non-surgical. Additionally, there must be some aspect of direct communication with an individual (or small group), whether in-person, by phone, by internet or other form. This ensures that participants are enrolled at the individual level, as we will not include interventions that are only performed at a community, campus, or other group level. Qualified comparison groups were usual care, pharmacological interventions, surgical interventions, or a lesser dose of the treatment. Interventions that include a drug or surgery in addition to the behavioral or psychosocial component were allowed, as long as the comparison group is not receiving the same behavioral or psychosocial intervention. The decision to use this broad definition of “behavioral or psychosocial intervention” was made to ensure a representative sample of these interventions in the literature. Through a preliminary look at the methods section of several studies in our search results, we observed a lack of detail in the reporting of intervention components, with some studies referring to previously published non-primary reports for intervention details. With the large scope of our review and limited time, it was not feasible for our team to look beyond the primary article to determine whether the intervention fit the behavioral or psychosocial criteria during the selection process. We did not want reporting quality to affect our selection of studies
How was “chronic condition” defined for the purpose of this review?
In this review we defined chronic condition using a list of 20 conditions complied by an MCC working group at the Office of the Assistant Secretary of Health within the US Department of Health and Human Services.* These chronic conditions meet the definition for chronicity, are prevalent and have potential to be modifiable by public health and/or clinical interventions.
These 20 conditions are:
Autism spectrum disorder
Chronic kidney disease
Chronic obstructive pulmonary disease
Congestive heart failure
Coronary artery disease
Dementia (including Alzheimer’s and other senile dementias)
Human immunodeficiency virus (HIV)
Substance abuse disorders (drug and alcohol)
Additionally, we included conditions defined in articles using general terms for chronic conditions (such as “chronic condition” or “chronic illness”) without specifying condition. General terms such as this without further clarifying information were rarely seen in selection criteria, but were used in both exclusion criteria and patient characteristics.
*Goodman RA, Posner SF, Huang ES, Parekh AK, Koh HK. Defining and Measuring Chronic Conditions: Imperatives for Research, Policy, Program, and Practice. Preventing Chronic Disease. 2013;10:E66.
How is this review different from previous work?
Although there have been multiple reviews that have assessed inclusion of individuals with MCC in RCTs (listed below), these reviews varied on the index conditions considered, the sampling techniques used, and the aspects of eligibility and inclusion of MCC assessed. Most reviews did not limit by intervention type, and there has not been a comprehensive review assessing the inclusion of individuals with MCC specifically in trials of behavioral and psychosocial interventions. Additionally, many previous reviews limited their sample to trials published in high-impact journals. These trials may have a different quality of trial design or reporting, and may not be representative of the inclusion and reporting of MCC in RCTs among trials that encompass the current literature.
The current review differs from previous reviews by (a) focusing solely on RCTs testing behavioral and/or psychosocial interventions; (b) considering a previously defined list of 20 chronic conditions chosen for their chronicity, prevalence, and potential to be modifiable by public health and/or clinical interventions; (c) using a comprehensive search strategy and sampling technique to produce a large representative subset of the literature across 15 years (2000-2014); (d) evaluating a wide range of variables related to trial design, trial quality, eligibility criteria, participant selection, and consideration of comorbidities in analysis; and (e) using best practices for systematic reviews including review of search results and selection of included studies and extraction of data by two independent readers.
Boyd CM, Vollenweider D, Puhan MA. Informing evidence-based decision-making for patients with comorbidity: availability of necessary information in clinical trials for chronic diseases. PLoS One. 2012;7(8):e41601. PMC3411714:
Jadad AR, To MJ, Emara M, Jones J. Consideration of multiple chronic diseases in randomized controlled trials. JAMA: the journal of the American Medical Association. 2011;306(24):2670-2672.
Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA : the journal of the American Medical Association. Mar 21 2007;297(11):1233-1240.
Schmidt AF, Groenwold RH, van Delden JJ, et al. Justification of exclusion criteria was underreported in a review of cardiovascular trials. Journal of clinical epidemiology. Jun 2014;67(6):635-644.
Zulman DM, Sussman JB, Chen X, Cigolle CT, Blaum CS, Hayward RA. Examining the evidence: a systematic review of the inclusion and analysis of older adults in randomized controlled trials. Journal of general internal medicine. 2011;26(7):783-790.
du Vaure CB, Dechartres A, Battin C, Ravaud P, Boutron I. Exclusion of patients with concomitant chronic conditions in ongoing randomised controlled trials targeting 10 common chronic conditions and registered at ClinicalTrials.gov: a systematic review of registration details. BMJ open. 2016 Sep 1;6(9):e012265.