Coherence is the essence of everything successful. Then we add integrity.

Curious look into quality of evidence produced in clinical trials

8. january 2017 at 20:34 | Veronika Valdova, ARETE-ZOE |  Medicine & Pharmacy
As of today, there are 187,607 interventional studies in the registry. This NIH-operated database is not the only one, but the most relevant and complete source on information on ongoing and completed clinical trials. The available information includes basic study descriptors such as phase, type, interventions, funding, and sponsors and co-sponsors; number of enrolled subjects and their demographic characteristics such as gender and age, and more detailed descriptors of study design such as masking, intervention model and allocation; as well as primary purpose and endpoint classification. This article is the first piece from a series of analyses of interventional trials on this database.

In interventional trials, theparticipants are assigned to receive one or more interventions or none at all, as assigned in the protocol. The objective is to gather and evaluate data from comparable groups of participants who only differ in the intervention they receive so that researchers can evaluate effects of these interventions on biomedical or health-related outcomes. In general, interventions can be diagnostic, therapeutic, or behavioral.

Level of Evidence
For a biomedical researcher, the level of evidence produced in clinical trials is essential to determine the overall impact of the study on benefit-risk assessments of the studied interventions, treatment guidelines, recommendations for reimbursement, and other assessments. Most organizations have a structured and comprehensive way of assessing scientific evidence that is coming in. However, there is no universal standard or generally accepted guidance for doing so, and these evaluations differ from one organization or journal to another.

Level I evidence includes:
  • High quality randomized trial or prospective study;
  • Testing of previously developed diagnostic criteria on consecutive patients;
  • Sensible costs and alternatives;
  • Values obtained from many studies with multiway sensitivity analyses;
  • Systematic review of Level I RCTs and Level I studies.
Some past scandals illustrate the extent of problem very clearly: scientific opinions on the value, power and level of evidence produced by the RECORD study meant the difference between acceptance and rejection of GSK's drug Avandia. Assessment of quality of evidence produced by a study has real-life consequences, both medically and financially.

Unfortunately, does not provide own (or authorized third party) assessment of evidence produced by studies included in the registry, and NIH leaves it solely to the assessor to decide how valuable each individual study is in the context of all other available information.

Processing the dataset before analysis included deletion of superfluous columns, standardization of null values, and deletion of trials that are scheduled to start after 2016. The material therefore does not serve as prediction but only review of studies that were already started.

Enrollment figures are skewed by very small number of studies with very high number of participants (millions of subjects). Three studies listed the total number of participants at 99,999,999 (a 100 million). These anecdotal figures make the true trends difficult to assess. Moreover, the number of planned and enrolled subjects can differ significantly so reliability of this data greatly depends on stage of the trial (plan, estimate, in progress, or completed and updated).

IBM Watson was used to analyze the datasets and create the visualizations. Detailed insight into individual types of interventions will follow in the next few weeks.

Type of Interventions
About a half of all clinical trials studied drugs as the intervention in question. Biologics, devices, procedures, and other type of interventions account for most of the remaining 50%. Relatively small number of interventions concerned radiation and dietary supplements. Breakdown by phase shows similar distribution by each phase: majority of interventions for trials in phase I to IV are drugs, followed by biologics, devices and procedures. Relatively high number of trials does not have any phase stated; for these the interventions fall almost evenly fall into categories "other", "behavioral", "device", "drug" and "procedure".

As mentioned previously, about one third of clinical trials have no phase stated. Early stage trials (phase I and II) create one third, and the remaining third goes to phase III and IV studies. One would expect that the highest number of enrolled subjects would be in late stage and post-marketing studies but that's not the case: the few previously mentioned studies with very high declared number of subjects are Phase I and undeclared.

Primary purpose
Nearly a half of studies are conducted for treatment purposes. About 40% of all studies have no primary purpose stated.

The highest share of studies, about a half, is funded by entities categorized as "other", even more if we include "other" in cooperation with other entities. This is a very broad category of academic institutions, individuals, and community-based organizations including hospitals. About one third of all trials is funded by industry.

Endpoint classification
The majority of all studies are conducted to satisfy safety and efficacy-related questions. More than a third of studies have no endpoint stated.

Quality attributes
The main quality attributes readily available for scrutiny are characteristics of study design: masking, allocation, and intervention model.

Allocation is a clinical trial design strategy used to assign participants to an arm of a study. In a Randomized Controlled Trial, the participants are assigned to intervention groups by chance. If the trial is non-randomized, the participants are expressly assigned to intervention groups through a non-random method, such as physician choice. Allocation is not relevant for single arm studies where all participants receive the same treatment. The gold standard scientific evidence is produced through randomized controlled trials.

Masking is a design strategy in which one or more parties involved in the trial, usually the investigator or participants, do not know which participants have been assigned which interventions. The main purpose of masking is to prevent placebo effect, post-randomization confounding bias, selection bias, or group differences n loss to follow-up, and information bias - that is bias due to differences in reporting of symptoms.

Open label study describes a clinical trial in which masking is not used, and all parties involved in the trial know which participants have been assigned which interventions.

In single blinded studies, one party involved in the clinical trial, typically the investigator or participants, does not know which participants have been assigned which interventions.

In double-blinded studies, two or more parties do not know which participants have been assigned which interventions. Typically, the parties include the investigator and participants, but sometimes also the assessor or caregiver.

Finally, intervention model describes design of the strategy for assigning interventions to participants in a clinical study.
  • Parallel Study Design means that two or more groups of participants receive different interventions "in parallel".
  • In a Cross-Over Study design, groups of participants receive two or more interventions in a particular order, so the participants "cross over" from one drug to the other.
  • Single Group Study Design means that all participants receive the same intervention.
  • In Factorial Study Design, groups of participants receive one of several combinations of interventions so during the trial, all possible combinations of the two drugs are given to different groups of participants.
RCTs, the gold standard
In Randomized Controlled Trials, direct comparison is made between two or more treatments groups. Randomization eliminates baseline differences in risk between treatment and control groups. Randomization should make all groups similar in terms of the distribution of risk factors whether these risks are known or unknown. The larger the groups, the greater the probability of equal baseline risks. However, participants in randomized controlled trials are often not representative of the target population, which introduces selection bias and generalizability.

Data available in the registry cannot satisfactorily answer this poignant question. It would make life of health professionals much easier, if study results and level of evidence became standard part of all trial reports.

New comment

Log in
  Don't you have your own web yet? Create it for free on

Actual articles