One of the co-authors, Roberta Scherer, is a PLoS ONE Editorial Board member. This does not alter the authors’ adherence to all the PLoS ONE policies on sharing data and materials.
Conceived and designed the experiments: RWS PS KD AE. Performed the experiments: RWS PS KD AE. Analyzed the data: RWS. Contributed reagents/materials/analysis tools: KD AE. Wrote the paper: RWS PS KD AE.
To reduce publication bias, systematic reviewers are advised to search conference abstracts to identify randomized controlled trials (RCTs) conducted in humans and not published in full. We assessed the information provided by authors to aid identification of RCTs for reviews.
We handsearched the Association for Research in Vision and Ophthalmology (ARVO) meeting abstracts for 2004 to 2009 to identify reports of RCTs. We compared our classification with that of authors (requested by ARVO 2004–2006), and authors’ report of trial registration (required by ARVO 2007–2009).
Authors identified their study as a clinical trial for 169/191 (88%; 95% CI, 84–93) RCTs we identified for 2004, 174/212 (82%; 95% CI, 77–87) for 2005 and 162/215 (75%; 95% CI, 70–81) for 2006. Authors provided registration information for 107/172 (62%; 95% CI, 55–69) RCTs for 2007, 103/153 (67%; 95% CI, 60–75) for 2008, and 126/171 (74%; 95% CI, 67–80) for 2009. Most RCT authors providing a trial register name specified ClinicalTrials.gov (276/312; 88%; 95% CI, 85–92) and provided a valid ClinicalTrials.gov registration number (261/276; 95%; 95% CI, 92–97). Based on information provided by authors, trial registration information would be accessible for 48% (83/172) (95% CI, 41–56) of all ARVO abstracts describing RCTs in 2007, 63% (96/153) (95% CI, 55–70) in 2008, and 70% in 2009 (118/171) (95% CI, 62–76).
Authors of abstracts describing RCTs frequently did not classify them as clinical trials nor comply with reporting trial registration information, as required by the conference organizers. Systematic reviewers cannot rely on authors to identify relevant unpublished trials or report trial registration, if present.
Randomized controlled trials (RCTs) that are used to inform systematic reviews are typically identified through searching electronic databases, such as PubMed, EMBASE, and the Cochrane Central Register of Controlled Clinical Trials. Because only 60% of trials initially reported as conference abstracts are ever published in full
Clinical trial registration was proposed in the 70’s and 80’s, in part as a mechanism of combating reporting biases [2.3], and the notion slowly gained ground in the 90’s and 2000’s
With 100 to 200 abstracts describing clinical trials published annually, the Association for Research in Vision and Ophthalmology (ARVO) annual meeting is an important source of reports of RCTs for the Cochrane Eyes and Vision Group (CEVG). We have searched the ARVO meeting books or the ARVO online abstract repository from 1990 forward for reports of RCTs and controlled clinical trials, to contribute to CEVG’s specialized database of trial reports. Because clinical trials represent only 2 to 3% of ARVO abstracts published each year, and completing the searches requires about 45 hours per year
“Is the research presented in your abstract a human clinical trial? [Yes] [No] (see hyperlink definition)
“Is the research presented in your abstract a human controlled clinical trial?” [Yes] [No] (same hyperlink definition as 2004)
In 2005, ARVO adopted a policy requiring registration of controlled trials in an electronically searchable, publicly available register before submission of abstracts to the ARVO annual meeting or articles to the associated journal,
“Please answer the following information below regarding Clinical Trials Registration. (Required)
Does the research presented in your abstract report on a clinical trial (refer to item #4 of the “ARVO Statement on Registering Clinical Trials” [hyperlink] and/or FAQs [hyperlink]) about the ARVO policy. [Yes] [No]”
Hyperlink to item#4 “
A hyperlink to “Clinical Trial Explanation” was also included.
“Please answer the following information below regarding Clinical Trial Registration. (Required)
Clinical trials require registration with a publicly accessible clinical trials registry that is approved by the World Health Organization (WHO). All abstracts that describe results from a clinical trial must include the registry site and registration number of the trial. To determine if the study results presented in your abstract are from a clinical trial, consider the following questions and refer to the “ARVO Statement on Registering Clinical Trials” [hyperlink] and/or FAQs [hyperlink]) about the ARVO policy.
1. Does this study involve a therapeutic intervention in human subjects? (The intervention may be of any kind, e.g., medical, surgical/laser, or psychological/sociological.)
2. Is the study prospective?
If the answer is “No” to either question, then the study does not meet the current definition of a clinical trial, and does not need to be registered. Select “No” below. If the answer is “Yes” to both questions, then the study does meet the definition of a clinical trial, regardless of the number of subjects involved or whether it involves comparison groups (i.e., different doses of a drug, or treatment and control groups) and must be registered. Select “Yes” below, then select the registry site and enter the corresponding registration number.
*Does the study meet the definition of a clinical trial? [Yes] [No]”
The drop-down was revised to include the following trial registers:
Our overall study objective was to assess the reliability of information provided by ARVO abstract authors that might be used to aid in identification of relevant abstracts for systematic reviews. We evaluated whether authors correctly classify abstracts describing RCTs as clinical trials, and as an adjunct to this, determined where investigators are registering their RCTs.
We searched the ARVO annual meeting abstract book (in 2004 we searched a CD-ROM; in 2005 through 2009 we searched online at
Following each annual meeting, the meeting organizers provided us with an electronic list of abstract program numbers that were identified by the author(s) as a “human clinical trial” (2004) or “human controlled clinical trial” (2005 and 2006) report (see Box for definitions). For the 2007 meeting, the meeting organizers provided an electronic list of the program number, trial register name, and registration number for all abstracts with trial registration information. For the 2008 and 2009 meetings, the organizers provided a hardcopy lists that included this information. We searched for and obtained paper copies of the abstracts on the organizers’ lists that we had not already identified as RCTs (2004 through 2009).
We compared the list of abstracts provided by the meeting organizers with the abstracts that we had previously classified as RCTs. Any abstract on the list that the handsearcher had not classified as an RCT was re-evaluated. If the handsearcher agreed that the report described an RCT, then that abstract was included in the reference standard for that year.
Thus, for each year, there were two groups of studies:
Abstracts in the reference standard:
$ Classified as a clinical trial by the author and as an RCT by the handsearcher;
$ Not classified as a clinical trial by the author, but classified as an RCT by the handsearcher.
Abstracts not in the reference standard:
$ Classified as a clinical trial by the author, but not classified as an RCT the handsearcher.
$ Not classified as a clinical trial by the author, nor classified as an RCT by the handsearcher.
A second handsearcher (PCS) reviewed the classification of 100% of abstracts that were included in the reference standard, i.e., abstracts that were classified by the handsearcher as an RCT. A 10% sample of abstracts classified by the author as a clinical trial, but not as an RCT by the first handsearcher, and a 5% sample of abstracts not classified as a clinical trial by the author nor as an RCT by the handsearcher were reviewed by another handsearcher (KD or AE). Abstracts in question were re-reviewed by the lead author for a decision or by both readers to arrive at consensus.
We classified the type of organization recorded in the trial registration field (trial register, ethics board/institutional review board, regulatory agency, local authority, or other) for conference abstracts presented in 2007, 2008, and 2009. For trials in the reference standard that were reported by the author as registered at ClinicalTrials.gov, we verified the information by entering the trial registration number provided in the ClinicalTrials.gov search webpage [
We entered into an Access database the program number and year of presentation of each abstract describing an RCT that we identified by handsearching the conference books, and imported the program number and year of presentation received from the meeting organizers for the abstracts from the years 2004 to 2006 as well as the program number, year of presentation, register name, and registration number for abstracts presented in years 2007 to 2009. We calculated the proportion of RCTs in the total number of abstracts presented at the conference for 2005–2009, and the proportion of RCTs in our reference standard that were correctly identified by the author, performing separate comparison analyses by year. We tabulated information recorded in the trial register fields, including trial register “name” and valid trial register number (yes/no). Results are presented as point estimates with 95% confidence intervals (CI). Two individuals performed all calculations independently.
The first handsearcher classified 1,121 of the abstracts presented at ARVO from 2004 to 2009 as RCTs. A second person read all 1,121 abstracts and classified 1,101 as RCTs, 10 as questionable, and 10 as “not RCT”. After re-review the 2 readers reached consensus on the 20 articles where there was initial disagreement;, resulting in 1,114 abstracts that were included in the reference standard as RCTs.
The first handsearcher classified 1529 abstracts, originally categorized by the author as a clinical trial, as “not RCTs”. A second person read a 10% sample (153) of these abstracts and classified 3 as RCTs and 150 as “not RCTs.” The first handsearcher, re-reviewed the 3 abstracts classified as RCTs by the second reader, and decided to keep the original classification of “not RCT.”
The first handsearcher classified 32,565 abstracts, which had not been classified as a clinical trial by the author, as “not RCT.” A second person read a 5% sample (1628 abstracts), and classified all as “not RCT.”
The number of RCTs presented annually at the 2004 to 2009 ARVO meetings and in the reference standard ranged from 153 to 215. Authors identified 169/191 (88%; 95% CI, 84–93) of all RCTs we identified for 2004, 174/212 (82%; 95% CI, 77–87) for 2005, and 162/215 (75%; 95% CI, 70–81) for 2006, by checking the “human clinical trial” or “human controlled clinical trial” box. A lower proportion of RCTs were identified by authors in 2007 (107/172; 62%; 95% CI, 55–70), 2008 (103/153; 67%; 95% CI, 60–75), and 2009 (126/171; 74%; 95% CI, 67–80), years requiring trial registration information rather than a checked box to designate a clinical trial (see
Year of meeting | Reference standard RCTs | Reference standard RCTs classifiedby author as clinical trial | Abstracts classified by author as clinical trial | ARVO abstracts presented |
No. | No. (%) | No. | No. | |
|
191 | 169 (88) | 634 |
5,610 |
|
212 | 174 (82) | 487 |
5,732 |
|
215 | 162 (74) | 454 |
5,920 |
|
172 | 107 (62) | 258 |
6,044 |
|
153 | 103 (67) | 252 |
6,122 |
|
171 | 126 (74) | 287 |
5,787 |
|
1,114 | 841 (75) | 2,372 | 35,215 |
Box checked “Yes” for “human clinical trial” (n = 634).
Box checked “Yes” for “human controlled clinical trial” (n = 941).
Information recorded in trial registration box (n = 797).
Does not include 576 abstracts that were withdrawn and are not included in the analyses.
Year | CEVG classification of abstracts identified by author as controlled trial | ||||||
RCT | Non-randomized trial | Not human | Un-controlled case series | Comparison by group characteristic | Other |
Total | |
No. (%) | No. (%) | No. (%) | No. (%) | No. (%) | No. (%) | No. | |
|
169 (26.7) | 59 (9.3) | 13 (2.0) | 195 (30.8) | 140 (22.1) | 58 (9.1) | 634 |
|
174 (35.7) | 58 (11.9) | 17 (3.5) | 139 (28.5) | 87 (17.9) | 12 (2.5) | 487 |
|
162 (35.6) | 60 (13.2) | 20 (4.4) | 130 (28.6) | 59 (13.0) | 23 (5.1) | 454 |
|
107 (41.5) | 18 (7.0) | 14 (5.4) | 96 (37.2) | 20 (7.8) | 3 (1.2) | 258 |
|
103 (40.9) | 13 (4.6) | 2 (0.8) | 87 (34.5) | 34 (13.5) | 13 (5.2) | 252 |
|
126 (43.9) | 18 (6.3) | 3 (1.0) | 97 (33.8) | 19 (616) | 24 (8.4) | 287 |
|
841 (35.5) | 226 (9.5) | 69 (2.9) | 744 (31.3) | 359 (15.1) | 133 (5.6) | 2,372 |
Box checked “Yes” for “human clinical trial”.
Box checked “Yes” for “human controlled clinical trial”.
Information recorded in trial registration box.
Includes: description of study methods (e.g., methods for measuring outcomes), systematic reviews, theoretical models, studies on correlation between test methods, and studies with historical controls).
Authors provided trial registration information at abstract submission for 797 abstracts (see
Information provided in registration box 2007–2009 | RCTs | Non-RCT | Total |
No. (%) | No. (%) | No. (%) | |
Trial register name | 312 (92.9) | 366 (79.4) | 678 (85.1) |
Ethics board | 5 (1.5) | 24 (5.2) | 29 (3.6) |
Local authority | 2 (0.6) | 32 (6.9) | 34 (4.3) |
Regulatory agency | 3 (0.8) | 4 (0.9) | 7 (0.9) |
Pending or reason for no registration | 1 (0.3) | 0 (0) | 1 (0.1) |
Other | 13 (3.9) | 35 (7.6) | 48 (6.0) |
|
|
|
|
Almost all RCT authors who provided a trial register name specified ClinicalTrials.gov (see
Trial register name | 2007 | 2008 | 2009 | Total |
No. (%) | No. (%) | No. (%) | No. (%) | |
Reference standard | 172 (100) | 153 (100) | 171 (100) | 496 (100) |
ClinicalTrials.gov(valid number) | 74 (43) | 86 (56) | 101 (59) | 261 (53) |
ISRCTN | 4 (2) | 4 (3) | 13 (8) | 21 (4) |
EudraCT | 2 (1) | 2(1) | 1 (0.6) | 5 (1) |
ANZCTR | 1 (0.6) | 2 91) | 2 (1) | 5 (1) |
Trialregister.nl | 0 | 0 | 1 (0.6) | 1 (0) |
Umin.ac.jp | 0 | 2 (1) | 0 | 2 (0.5) |
UK national research register | 1 (0.6) | 0 | 0 | 1 (0) |
European clinical trials database | 1 (0.6) | 0 | 0 | 1 (0) |
|
|
|
|
|
Abbreviations used: ISRCTN: International Standard Randomised Controlled Trial Number Register, EudrACT: European Union Drug Regulating Authorities Clinical Trials, ANZCTR: Australian New Zealand Clinical Trials Registry, Trialregister.nl: Nederlands Trial Register, Umin.ac.jp: University Hospital Medical Information Network.
We were disappointed that asking authors to identify their abstracts as describing trials is not reliable, since it could have helped to avoid laborious handsearching of conference abstracts required to identify all RCTs for systematic reviews
One of the reasons for the development of trial registers is to provide information about all initiated trials, including those remaining unpublished. Although registration does inform the public about the existence of trials, most investigations to date have demonstrated that registration details (e.g., study design, protocol and contact information) are less than optimal for inclusion in systematic reviews
Alternatively, authors may have registered their trials, but may not have entered the required trial registration information on the abstract submission form. Lack of compliance with abstract submission requirements related to trial registration would not be surprising, as compliance with required trial registration is also problematic for journal articles
The fact that authors do not always recognize their trials as RCTs and the apparent lack of trial registration has broad implications for identifying all the evidence for informing systematic reviews and healthcare decision making. If authors have no intention of publishing their findings and registration is not required by law
Our findings are limited to abstracts submitted to a single conference from 2004 through 2009 and may not apply to other years or areas of clinical research. The relatively low rate of correct identification of RCTs across the years searched implies that there is an ongoing problem with author classification of RCTs. Although we observed a high “false positive” rate for trial registration, this result may be due to the fact that registration is required for many types of “clinical trials” not just RCTs. We did not expect all abstracts with trial registration to be RCTs, but we did expect that all RCTs would be registered. We had hoped that a set of abstracts with trial registration would provide us an enriched and comprehensive source of RCTs to reduce the time and effort required to search the conference abstracts.
Our findings lead us to be somewhat pessimistic about authors being able to identify their own studies as randomized clinical trials. In addition, RCT investigators may not be registering their trials or reporting trial registration. Thus, it is unlikely that systematic reviewers would be able to use the author-classification of study design to identify ARVO abstracts describing RCTs.