Introduction

Early in the COVID-19 pandemic, massive research efforts were mobilised to identify potential treatments for the novel disease. This resulted in an unprecedented number of clinical trials being initiated for a single disease during a short period of time.

Prior to the pandemic, studies had demonstrated that a significant proportion of clinical trials are terminated prematurely with insufficient sample size or remain unpublished, leading to considerable research waste.1 Research waste in clinical trials also stems from inadequate research questions, flawed study design and shortcomings in trial conduct, including alteration of the primary outcome from protocol registration to publication, which results in misleading conclusions, distortion of the evidence-base and further research waste.2 3

Already in the first months of the pandemic, concerns were raised that the rushed and uncoordinated research efforts led to the initiation of studies with limited potential to improve the evidence base for the treatment of COVID-19.4 5 It was noted that many of the studies were small and fragmented: numerous studies investigated the same interventions and had low expected strength of evidence, already when being designed and planned.

While research waste associated with clinical trials on COVID-19 was subject to much attention early in the pandemic, what happened to these trials after their registration remains unknown. This is important to investigate as such information can be used to improve research coordination and response to future pandemics and health crises with acute needs to gather useful evidence on treatment effects.

In this study, we described all clinical trials registered on ClinicalTrials.gov during the first 6 months of the COVID-19 pandemic. We assessed if and where the results had been reported at around 3 years after study registration, completion and discontinuation rates of the trials, reasons for discontinuation, achieved enrolment and any changes made to the primary outcome between trial registration and the reporting of study results.

Methods

This was a cohort study of clinical trials registered on ClinicalTrials.gov.

Search strategy

We searched ClinicalTrials.gov on 15 May 2023 for interventional studies, first registered between 1 January 2020 and 1 July 2020 using the term ‘Covid-19’. The search returned all registered studies with search terms linked to ‘Covid-19 on ClinicalTrials.gov (SARS-CoV2 Infection; Covid-19 infection; Covid-19; Coronavirus disease 2019; SARS CoV 2 Infection; Covid-19 Virus Infection; Coronavirus Disease 19; Coronavirus Disease-19; 2019 Novel Coronavirus Disease; SARS-CoV-2 Infections; Covid-19 Virus Disease; 2019 Novel Coronavirus Infection; SARS-CoV2 Disease; 2019 nCoV Infection; Severe Acute Respiratory Syndrome Coronavirus 2 Infection; 2019 nCoV Disease; Disease caused by SARS-CoV-2; coronavirus 2019-nCoV; SARS Coronavirus 2 Infection; coronavirus 2019-nCoV infection; Disease caused by 2019-nCoV).

From this search, we extracted all structured data fields. These entries were manually assessed to exclude clinical trials that were uncontrolled (comparing fewer than two groups), did not assess a treatment intervention aimed at improving outcomes for patients with COVID-19 (eg, assessments of diagnostic tools or interventions aimed at preventing COVID-19 infection) or were not prospective. We used a pragmatic approach to study exclusion: we screened all trials and noted the first exclusion criterion/criteria that was observed. Once a trial met an exclusion criterion, we did not assess whether it fulfilled other exclusion criteria. Excluded register entries are shown in online supplemental file 1.

Clinical trial characteristics

Three investigators (PT, AJ, EW) extracted information about clinical trial characteristics using information from ClinicalTrials.gov and assessed the reported results as described below. Characteristics and results of 100 studies were assessed in duplicate, and disagreements were resolved by discussion; the remaining studies were assessed by a single investigator, with periodic duplicate agreement checks.

Definitions of and rationales for the variables assessed in our analyses are shown in online supplemental table 1. We assessed clinical trial characteristics including planned study completion date; location (categorised as the USA; Europe; China; Other Asia; South America, Mexico and the Caribbean; Africa; Canada; Oceania; multinational; not specified); industry funding (any registered industry funding; no registered industry funding); planned study enrolment (target number of enrolled participants according to the first registration of the trial protocol); type of study population (hospitalised patients; non-hospitalised patients; both hospitalised and non-hospitalised patients; other); randomised intervention (yes; no); type of comparator/control (placebo; standard care; another drug or dose of the same drug); number of interventions compared; number of primary outcomes and type of primary outcome (using binary non-mutually exclusive variables for: composite outcome, all-cause mortality, mechanical ventilation, clinical improvement, laboratory-based outcomes, hospitalisation and other type of outcome). We also investigated the number of unique interventions assessed in the trials. For each trial, we reviewed the study design section on ClinicalTrials.gov to confirm the actual interventions under trial, as those listed in intervention fields sometimes differed from the ones used in the trial. We categorised medications based on the generic name of the drug.

Assessment of reported results

Reported results of the clinical trials were assessed by systematically searching for the trial registration number on PubMed, medRxiv, Google (assessing the first 20 search results) and Google Scholar; the name of the trial’s principal investigator in PubMed; and results reported on ClinicalTrials.gov. Results reported through 15 May 2023 were considered.

Status of the clinical trial and results reported in publications in scientific journals, preprints and on ClinicalTrials.gov were assessed. We categorised clinical trials based on whether the results had been reported (further categorised into those published in a scientific journal, those published as preprint but not in a scientific journal and those reported only on ClinicalTrials.gov) or were not reported (further categorised into those that had been withdrawn from ClinicalTrials.gov, those with other registered trial statuses and those that had been published in a scientific journal and subsequently retracted).

For results reported on ClinicalTrials.gov, all registered results were considered regardless of the quality control status on ClinicalTrials.gov. Trial results that were not available on ClinicalTrials.gov despite being registered as submitted were considered not reported. Trials with reports that only included the characteristics and outcomes of the treatment group but not the control group were considered reported.

Enrolment of patients in trials with reported results was assessed. Trials that had achieved less than 90% of the target sample size6 specified in the first protocol registration were considered discontinued. If a trial was identified as adaptive (eg, the RECOVERY and DISCOVERY trials), the achieved enrolment was defined as the cumulative enrolment across various published studies. For non-adaptive trials for which two or more publications were found under the same National Clinical Trial (NCT) number, the study report with the largest enrolment corresponding to the trial protocol was considered. When multiple trial registrations from more than one study centre led to a single published study, each trial registration was evaluated separately; the achieved enrolment was assessed based on the number of patients recruited at the study centre that registered the study. Reported reasons for discontinuation were assessed (slow recruitment; findings from interim analyses; external evidence; other reasons; no reported reason); a trial could report more than one reason for discontinuation.

For the analyses of outcome switching in the reported trials, we first extracted the primary outcome reported in the publication, the preprint or on ClinicalTrials.gov. If not explicitly stated in the results section, we used the outcome from the methods section or power calculation. Comparing the reported primary outcome with the primary outcome listed in the initial ClinicalTrials.gov registration, we categorised trials with mismatches in the reported vs initially registered primary outcome (including change of the outcome, as well as the addition or omission of registered outcomes), as having undergone outcome switching.7

Following reviewer comments, we conducted a post hoc review of all trials categorised as having undergone outcome switching. An investigator (NA) re-evaluated whether changes to the primary outcome were made and if those changes were disclosed in the trial report. We also reviewed all versions of the trial registration to assess when the switch to the reported outcome had occurred. For trials that had not registered the switch, the date of outcome switching was set to the date of trial reporting (online supplemental table 1). We also assessed the start and end dates of patient recruitment as reported in the trial paper or preprint. For trials that did not report these dates, we used the trial start date and completion date registered in the latest trial registration on our search date. Start and end dates of patient recruitment were used to assess the time from recruitment start to outcome switching and the proportion of trials that changed the primary outcome before start of patient recruitment and after the end of patient recruitment, respectively. Finally, we assessed whether the outcome switching was disclosed in the trial paper or preprint.

Statistical analysis

We described trial characteristics in the total sample and separately for trials whose results had been reported vs not reported. We then described where trial results had been reported, completion status and whether the primary outcome had been changed after trial registration. As we noted that few trials had been completed, published in a scientific journal and enrolled a large study population, we decided post hoc to describe the characteristics of trials that were (a) completed and published in a scientific journal and (b) completed, published in a scientific journal and enrolled more than 100 participants; this cut-off for enrolment was used because trials with smaller sample size are considered to rarely yield informative results.8 We also assessed the number of trials which were completed, published in a scientific journal, enrolled more than 100 participants and had not changed its primary outcome.

Patient and public involvement

No patients were involved in setting the research question nor in the design, conduct or interpretation of the study.

The data used in the analyses are provided in online supplemental file 2.

ResultsClinical trial characteristics

Of the 1317 studies identified through the search on ClinicalTrials.gov, 542 were excluded due to being uncontrolled, not assessing a treatment intervention for patients with COVID-19 or not being prospective. 775 clinical trials were included in the study. Nearly all trials (n=758 (98%)) had a planned completion date before 15 May 15 2023 (last day for which reported results were considered in our search); 227 trials (29%) were based in the USA, and 80 trials (10%) were multinational; 584 (75%) included hospitalised patients, and 315 (41%) were placebo-controlled. The total planned enrolment across all trials was 238 933 (median (IQR) 120 (60, 304) patients); 470 (61%) planned to enrol 100 or more patients (table 1).

Table 1

Characteristics of clinical trials of COVID-19 registered at ClinicalTrials.gov between 1 January 2020 and 1 July 2020

The trials assessed 355 different interventions. Hydroxychloroquine was the most investigated intervention (n=73 (10% of all interventions)).

Reporting of results

Results had not been reported for 420 (54%) of the trials: 75 (10% of total) had been withdrawn from ClinicalTrials.gov, while 338 (44%) had other registered trial statuses on ClinicalTrials.gov, and 7 (1%) had been published in a scientific journal and subsequently retracted.

Results were reported for 355 (46%) trials: 283 (37% of total) were published in a scientific journal, 13 (2%) were reported as preprints but not in a journal, and 59 (8%) were only reported at ClinicalTrials.gov (figures 1 and 2).

Figure 1

Completion, discontinuation and reporting of results for clinical trials of COVID-19.

Figure 2

Venues of reporting for reported clinical trials of COVID-19.

Trial completion and changes in the primary outcome

A total of 95 332 patients had been enrolled in the reported trials (table 1). Of the trials with reported results, 186 (24% of total) were completed, and 169 (22%) were discontinued. A post hoc analysis of the discontinued trials showed that only two studies (1% of discontinued trials) would have been recategorised from ‘discontinued’ to ‘completed’ if the latest enrolment target registered before the start of study recruitment had been used instead of the first registered enrolment target. The median (IQR) enrolment was 125.5 (64, 350) in the completed trials and 85 (28, 191) in the discontinued trials (table 2). The discontinued trials had a total enrolment target of 86 276 patients and enrolled 28 623 patients. Slow recruitment (n=66, 39% of discontinued trials), findings from interim analyses (n=33, 20%), external evidence (n=20, 12%) and other reasons (n=22, 13%) were reported as reasons for discontinuation; n=50 (30% of discontinued trials) did not report a reason.

Table 2

Characteristics of discontinued and completed clinical trials of COVID-19 with reported results

Of the 355 trials with reported results, 117 (33%) trials had changed their primary outcome after the initial registration on ClinicalTrials.gov (69 (37%) of completed trials and 48 (28%) of discontinued trials) (tables 1 and 2). Only three (2%) of the trials which had changed the primary outcome had registered their outcome switch before starting recruitment. Of the 93 trials that were published in a preprint or scientific journal and had switched the primary outcome, 69 (74%) did not disclose the switch in the publication (table 1).

Completed trials published in a scientific journal

Of the 186 completed trials, 157 (20% of total) were published in a scientific journal. In these published trials, the median (IQR) enrolment was 149 (72, 400), 105 trials had enrolled 100 or more patients, and 103 had not changed the primary outcome between trial registration and publication. In total, 63 of the trials had been completed, published in a scientific journal, enrolled 100 or more patients and not changed the primary outcome; 22 of these trials were placebo-controlled (table 2).

Discussion

More than half of the 775 clinical trials registered on ClinicalTrials.gov during the first 6 months of the COVID-19 pandemic had no results reported or had been discontinued when examined at around 3 years after study registration. In total, 157 (20%) trials had been completed and published in a scientific journal, and 63 of them had enrolled 100 or more patients and not changed the primary outcome after trial registration. A third of trials whose results had been reported had changed their primary outcome after the first protocol registration, and most of those trials did not disclose the outcome switching when reporting the results.

In line with previous studies,9–11 we found that a substantial number of trials investigated the same interventions and suffered from low expected level of evidence due to not being placebo-controlled or having a low enrolment target. Our systematic assessment of the research output yielded from these trials expand the data regarding the eventual usefulness of clinical trials early in the pandemic: our findings show that the problems with study quality and enrolment continued also after trial initiation, with high rates of unreported results, study discontinuation and alterations in the primary outcome after study registration.

A previous study, the DIRECCT study,12 included COVID-19 trials registered in the WHO International Clinical Trials Registry Platform, with planned completion dates within the first 18 months of the pandemic (January 2020 through 30 June 2021), based on the last registered completion date at the time of their database search (1 July 2021). All included trials were considered ‘completed’, and when assessing the dissemination of results through 15 August 2021, the cumulative incidence of trial reporting was 32.8% at 12 months after the planned completion date. Direct comparisons with our findings are challenging, as we included trials regardless of planned completion dates or later withdrawal while the DIRECCT study only included trials with a planned completion date before 30 June 2021, excluding trials registered early in the pandemic that later postponed their planned completion dates, as well as withdrawn trials. This likely led the DIRECCT study to have a sample with higher reporting rates than in our study.

Importantly, while the DIRECCT study considered any trial as ‘completed’ after its planned completion date regardless of its enrolment or reporting status, trial completion in our study was defined as reaching at least 90% of the target enrolment.6 Trials not reaching their target enrolment were considered discontinued, while the completion status of unreported trials could not be determined. With almost 2 years of additional follow-up as compared with the DIRECCT study, the median (IQR) time from the planned completion date to our search for reported results was 871 (687, 986) days among the unreported trials. We therefore hypothesise that many of these trials were discontinued early before enough patients had been recruited to produce any useful data and that they will never be reported as a discontinued trial, let alone as a completed trial. Thus, we suggest that these trials should not be labelled as ‘completed’, as was done in the DIRECCT study. Future research may assess what happened to the unreported trials and thereby inform a much-needed consensus regarding the definition of trial completion in research on trial reporting.

Many of the clinical trials initiated early in the pandemic were hastily planned in a time of crisis. Moreover, the trials recruited similar patient populations and competition for research participants can lead to discontinued and underpowered studies.13 Therefore, it is possible that more research waste was generated from clinical trials of COVID-19 than from clinical trials on other outcomes planned and performed in a non-pandemic setting.5 However, the heterogeneity of study populations, target diseases, outcomes and follow-up time of clinical trials included in previous studies limit their direct comparability with COVID-19 trials.1 6 14–19 Moreover, although such comparisons facilitate interpretation of our findings against the background of insufficiencies in infrastructure, research coordination, investigator incentives and funding common to a broader set of clinical trials, it is important to consider the findings independently and in absolute terms. This is because the absolute research output in a pandemic is what matters for the improvement of evidence-based care and patient outcomes.

Nonetheless, in a previous study,20 55% of clinical trials registered at ClinicalTrials.gov between 2000 and 2019 were listed as completed in the database by 1 January 2020. However, as noted during our analyses and in previous studies, study completion as registered at ClinicalTrials.gov is highly unreliable,21 and this outcome needs to be assessed with a systematic search for reported results. Further, a study of 326 randomised clinical trial protocols approved in 2012 by ethics committees in Switzerland, the UK, Germany and Canada showed that 30% of the clinical trials were discontinued and 21% remained unpublished during 10 years of follow-up.14

Although previous studies from the last decades have shown varying rates of changes in the primary outcome after trial registration,22–24 most reported rates are similar to that observed in our study (35% of trials with reported results). Notably, some studies indicate a substantial decrease in the prevalence of primary outcomes changes during recent years,25 suggesting that post-registration changes to the primary outcome in our study of COVID-19 trials might be higher than that of other contemporary trials. Moreover, between 2000 and 2019, the median target enrolment of trials registered on ClinicalTrials.gov was just 60, with diminishing average study enrolment over time.20

Our findings are also partly in line with reports26 27 on the clinical trials conducted during the 2014–2015 Ebola outbreak. According to those reports, none of the clinical trials that investigated therapeutical interventions were able to reach useful conclusions about treatment efficacy. In some cases, the inconclusive trials may have hampered the pursuit of safe and effective treatments.27 Taken together, our and previous studies indicate a need for improved readiness to conduct clinical trials in response to rapidly emerging health threats. Such programmes and platforms had been prepared prior to the pandemic,28 resulting in some timely and large clinical trials being conducted early in the pandemic. However, our study shows that few of the trials conducted outside of such programmes yielded informative data, lending support to calls for broad, coordinated and strategic approaches to clinical research across healthcare and research institutions during health crises.29

Strengths of our study include the assessment of reported results, study enrolment and changes in the primary outcome using multiple data sources including publications, preprints and ClinicalTrials.gov, as compared with scraping of data from ClinicalTrials.gov,9–11 which has been shown to be unreliable.21 Moreover, investigators examined and compared the trial protocols with the reported results to extract key information and detect inconsistencies.

Our study has limitations. Registration of drug, device or biological interventions on ClinicalTrials.gov is required in the USA, but not globally. Although ClinicalTrials.gov is one of the major global databases for clinical trials, only half of non-US studies are estimated to be registered there,30 and of the interventional trials related to COVID-19 registered in the WHO International Clinical Trials Registry Platform through 15 May 2020, 54% were registered with ClinicalTrials.gov.31 In the DIRECCT study,12 over 70% of the clinical trials of COVID-19 included from the WHO International Clinical Trials Registry Platform had been registered on ClinicalTrials.gov. Moreover, as we aimed to assess the initial response to the COVID-19 pandemic by clinical investigators, we only included clinical trials registered during the first 6 months of the pandemic. Our findings may not be generalisable to more recently registered trials on COVID-19, some of which are still ongoing. Further, some of the clinical trials included in our study were adaptive platform trials. Although they assessed more than one intervention, they were counted as one observation in our analysis, and we may have underestimated their contribution to the overall research output of the COVID-19 trials.

In conclusion, we found that most clinical trials of COVID-19 registered at ClinicalTrials.gov during the first 6 months of the pandemic remained unreported or had been discontinued at around 3 years after registration. Many of the reported trials had enrolled few patients and changed the primary outcome after trial registration. Our findings add to the literature indicating a need for substantially improved organisation and coordination of research during future pandemics.