Comparison of media and academic attention of recently published positive and neutral or negative randomized cardiovascular clinical trials

Background: Citations are used to assess the importance of authors, articles and journals in the scientific community, but do not examine how they affect general public journal readership. The Altmetric Attention Score (AAS) is a new metric for measuring media attention of the published paper. Methods: We examined cardiovascular (CV) randomized clinical trials (RCTs), published in the 3 highest Web of Science Impact Factor journals (Journal Citation Reports 2019: category “Medicine, General & Internal”) and in the 3 highest Web of Science Impact Factor CV journals (Journal Citation Reports 2019: category “Cardiac & Cardiovascular Systems”), through the calendar year of 2017, 2018 and 2019. The primary outcomes were the assessment of the difference between number of citations and AAS among positive and negative CVRCTs. Results: Among the included 262 RCTs, more positive CVRCTs were published (p = 0.002). There was no significant statistical difference between the positive and negative trials, considering the number of citations (p = 0.61). Interestingly, positive trials had a tendency towards a higher AAS (p = 0.058). The correlation between the AAS and the number of citations was moderate positively correlated (ρ = 0.47, p < 0.001). Conclusion: We did not find any differences between CV RCTs with positive vs CV RCTs with negative results considering the number of their citations. A tendency towards a higher AAS among positive CV RCTs could indicate higher activity on social media regarding CV trials with positive results. A higher number of published positive CV RCTs among all published CV RCTs could indicate the presence of publication bias but further investigation of unpublished RCTs in trial registries (e.g., clinicaltrials.gov) is needed.


Introduction
Clinical research is the cornerstone of innovation in medical development. Randomized clinical trials (RCTs) form the basis of evidence-based practice [1,2]. Negative trials are considered those that fail to reject the null hypothesis for the primary endpoint, or rephrased, those trials that failed to achieve the prespecified aims of the investigators and/or sponsors. RCTs are accompanied with a substantial investment burden and they are often funded by pharmaceutical companies [3,4]. Therefore, it is common perception to view negative trials as a poor investment or even being valueless. However, this attribute is in fact incorrectly interpreted. Negative trials provide good evidence of which treatment lacks an effect, considering the prespecified primary outcome, or has an effect but in lesser extent than expected. Also, it is possible that inadequate design and poor conduction of the RCT could lead to negative results, even for interventions that have an effect [5]. Nevertheless, negative results prevent adoption of those interventions in clinical practice [6]. In addition, they lead to decreasing or aborting further research investment in those interventions. However, in 2017, WHO stated that 50% of clinical trials went unreported, often because of negative results [7]. These unreported results of trials could lead to a biased picture of the risk and benefits of medicines and medical devices and result in the use of suboptimal and even harmful products. In another statement the WHO calls for reporting results of older but still unpublished trials and outlines steps to improve linkages between clinical trial registry entries and their published results [8].
The importance of scientific research is gauged through the number of citations and the impact of the journal where the research was published [9]. Recently, we observed an increase in media attention regarding scientific research, which is getting distributed more progressively through a wide range of online sources [10]. It is anticipated that one third of the global population is going to use social media in 2021 [11]. One of the advantages of this new trend is that scientific research information is more accessible for the general public. Therefore, new metrics for the measurement of published papers impact on general public have been developed.
The Altmetric attention score (AAS) is one of these new metrics, and it offers a measure of the "media impact" of a scientific publication. It provides a numeric score derived from an algorithm, which allocates a score according to the detected online attention. This attention is weighted according to the source. For instance, a mention on news media generates a score of 8 while a mention on Facebook generates a score of 0.25 [12]. AAS is designed to complement traditional bibliometrics by the utilization of the information from digital platforms, and provision of immediate quantification of the impact of the scientific paper [13]. With the recent frequent utilization of AAS as a measure of scientific performance, we were interested in understanding the impact of this new score on scholarly article metrics. If a correlation between an articles AAS and it's number of citations exist, one could argue that by promoting research articles on social media and other digital platforms the scientific impact of a article could also be increased.
A recently published study (939 articles, 76% were observational studies), showed that among cardiovascular (CV) trials, the AAS had a moderate correlation with citation counts at 3 years [14]. Poor correlation between the AAS and the number of citations has been demonstrated in the cross-sectional analysis among high-impact general medicine journals [15].
The impact of positive and negative CV RCTs using citation data and the AAS and their correlation has not been evaluated. The aim of this study was to compare the publication impact of positive and negative CV RCTs through scholarly (number of citation) and new (AAS) bibliometrics measures. We also evaluated the possibility of the presence of publication bias associated with statistical significance among included trials.

Study design
Articles assessed for eligibility were full-length articles published in the 3 highest Web of Science (WoS) Impact ))). The query changed according to the searched journal and the analysed year. In addition, double checks were performed on the investigated journal website. In addition, manual review of one year (2017) of publications for each journal was performed to validate our search strategy. The manual review of 2017 showed a 100% retrieval rate for our search strategy. We have also performed a search of additional databases such as EMBASE, Science direct, Scopus, Google scholar.
We included RCTs involving patients with CV diseases and reporting CV outcomes as their primary outcome. Only the primary manuscript of the randomized trial was included, for which we considered the manuscript that reported results of the analysis for the protocol pre-specified primary outcome. Secondary manuscripts, such as the reported sub-analyses or follow up analyses were excluded. Articles that did not have a clearly defined primary outcome were not included in the analysis (Fig. 1).

Definitions
We have classified a trial as positive if the results reject the null hypothesis and support the alternative hypothesis for the designated pre-specified primary outcome according to the statistical design of the trial. All included positive trials should have defined a statistically significant (positive) result (p ≤ 0.05) for achieving a pre-specified primary outcome. A negative trial was considered as such, if results suggested that there was no statistically significant difference (p > 0.05) or statistical difference favoured control arm, for the observed effect between the experimental or control arm. Non-inferiority trials have been classified as positive if they achieved a prespecified aim for establishing non-inferiority. One should bear in mind that the null result does not suggest that the treatments are the same, only that there is no evidence that they were different. For the purpose of this analysis, we have considered both null and negative trials as failed experiments and attributed them as negative trials. For the trials with multiple primary outcomes, we have used the outcome with the smallest p value for the determination of the trial classification as positive or negative. Trials that have used Bayesian statistics for data analyses were classified as positive or negative according to the achievement of posterior probability for the prespecified success criteria.

Data abstraction
Two reviewers (DŠ, JK) assessed each article published in the aforementioned journals and included or excluded it according to the prespecified inclusion and exclusion criteria. The flowchart diagram of the study selection is presented in Fig. 1. In order to verify the degree of agreement, Cohen's kappa test was used with the following Kappa interpretation: <0 poor, 0.0-0.20 slight, 0.21-0.4 fair, 0.41-0.6 moderate, 0.61-0.80 substantial, 0.81-1.00 almost perfect [17]. In cases where there was a disagreement or difference regarding the included trials or extracted data between the two authors (DŠ, JK), it was resolved through a consensus after rechecking the source data and consultations with additional investigator (MS).
For each trial we extracted parameters, which were later used for the description of included trials and the analysis. The extracted data included first author, year and the journal in which the paper was published, number of randomized participants, type of the primary outcome (patient centred or surrogate), type of funding, design (e.g., noninferiority), intervention type (medication, interventional procedure, diagnostics or other), AAS, number of citations and the use of multiple primary outcomes. The funding source was determined according to the funding sources mentioned in the manuscript. It was classified as industrial if at least one of the stakeholders involved in the trial funding was from the industry setting. In case that the searched information was not provided in the main article, we searched the supplementary and online available data together with the corresponding ClinicalTrials.gov entries [18]. The date of the publication was classified according to the date on the article written on the journal website (Epub date was not considered). Patient-centred outcomes (PCOs) have been defined as variables that assess a direct clinical benefit and reveal a patient's feeling of well-being or survival.
The manuscript was investigated for number of citations in the WoS directory and the AAS. The AAS was assessed for each individual article via the Altmetric bookmarklet (available for open-source download from the altmetric website). Citation and the AAS data were updated on July 10th 2020.

Study outcomes
The primary outcome of our research was to investigate the difference in the number of citations and AAS among positive and negative CV RCTs.
The secondary outcomes included trends for publishing negative trials; noninferiority trials; industry funded trials and use of surrogate markers in RCTs investigating CV outcomes. In addition, we evaluated the correlation between the number of citations and the AAS among positive and negative CV RCTs.

Statistical analysis
The normality of the distribution of parameters was tested with the Shapiro-Wilk test. Since all continuous data were non-normally distributed, they were presented as median with interquartile range (IQR). Respectively, variables were compared using the Mann-Whitney U test, and the χ 2 test/Fisher's exact test were used where appropriate. Provided odds ratios (OR) were obtained by logistic regression. The correlation between numerical variables was assessed using Spearman rank correlation and described with the Spearman ρ correlation coefficient. The Cochran-Armitage test for trend was used for investigating any possible time trends in the publication of CV RCTs, regarding trial results or it's design. We have not performed any adjustments regarding multiple testing. Two-sided p values of equal or less than 0.05 were considered to be statistically significant. All statistical analyses were performed in the R studio program (2009-2019 RStudio, Inc. Version 1.2.1335, available at https://www.rstudio.com/products/rstudio/).

Results
A total of 262 RCTs were included, which involved patients with CV disease and reported CV outcomes as their primary outcome. The two independent reviewers (DŠ, JK) showed a high inter-rater reliability (Cohen's Kappa = 0.867, 95% CI 0.837-0.897) which indicates almost perfect agreement. The majority of the published trials investigated PCOs (76%). Median number of participants in trials was 575 (IQR 188-2204). More than half of CV RCTs were industry funded (57%) and investigated effect of a drug (56%) ( Table 1). There was no significant difference between positive and negative trials, considering the number of their citations (p = 0.66). However, positive trials had a tendency towards a higher AAS (p = 0.058). Positive trials had higher likelihood to be non-inferior by their design (OR 3.8, 95% CI 1.8-9.1, p = 0.001), and noninferiority trials were more of-ten funded by the industry (OR 2.7, 95% CI 1.3-5.8, p = 0.008), when adjusted for time since published. Interestingly, positive trials used surrogate markers more often as their primary outcome (OR 2, 95% CI 1.1-3.8, p = 0.027), when adjusted for time since published (Table 1) More positive CV RCTs were published (p = 0.002). We analysed the frequencies of p values between 0.01 and 0.09 and speculated whether they are uniformly distributed. However, we have noticed a decreasing trend of their distribution with a discrete step at value of 0.05 (Fig. 2), which is commonly considered a threshold for claiming statistical significance. This observation raises doubts regarding the presence of publication bias, which has already been described in the literature [19]. To really confirm those suspicions, further investigation of unpublished RCTs in trial registries (e.g., ClinicalTrials.gov) should be done.
Correlation between the AAS and the number of citations was moderately positively correlated (ρ = 0.47, p < 0.001), even after analysing positive and negative trials data separately. The correlation strength was similar when performing subgroup analyses considering the type of journal ("Medicine, General & Internal" journal vs "Cardiac & Cardiovascular Systems" journal), as well as study outcome (positive vs negative).
The highest correlation has been noticed for positive trials when log transformed and divided with time (months) (ρ = 0.57, p < 0.001) (Fig. 3).
There was no time trend associated with publication of industry funded (p = 0.37), negative (p = 0.34) and noninferiority trials (p = 0.39), or trials that used surrogate outcomes (p = 0.62). Only 3 out of 262 trials have used Bayesian statistics for data analysis.

Discussion
The results of our study suggest that there is no significant difference among recently published positive and negative CV RCTs regarding their impact measured through scholarly bibliometrics (number of citation). In other words, positive CV RCTs are not more cited then negative CV RCTs. However, we have noticed a tendency toward a higher AAS for recently published positive CV RCTs compared to the negative ones (p = 0.058). A possible reason may be a higher tendency of the general public to discuss results of positive trials on social media platforms. Another reason could be the selective social media promotion of positive cardiovascular trials by those with scientific, financial or other interests. However, this speculation needs to be further investigated.
The AAS and number of citations had a moderately positive correlation among CV RCTs which suggest that social media promotion of a CV RCT article could increase it's citation rate and scientific impact. The correlation strength did not change when performing subgroup analyses, considering the type of journal ("Medicine, General & Internal" journal vs "Cardiac & Cardiovascular Systems" journal) and study outcome (positive vs negative). This is in line with previous research that has investigated this type of correlation [15,20].
Analysing the AAS, we have noticed that values above median AAS were strongly associated with publications in "Medicine, General & General" journals (OR 14, 95% CI 7.8-25.9, p < 0.001). Possible explanation for this observation could be a strong twitter community of "Medicine, General & Internal" journals which can have huge influ-ence on the dissemination of research among social platforms. Also this contributes to the formation of the AAS by itself. A much higher proportion of CV RCTs (76%) published in "Medicine, General & Internal" journals than in "Cardiac & Cardiovascular Systems" journals investigated the effect of intervention on PCOs, which is not the case in some other medical settings such as oncology and hematology [20]. In addition, the majority of articles published in "Medicine, General & Internal" journals were funded by industry (p = 0.049), which can be explained by a higher financial requirement for the conduction of larger trials which are commonly published in those journals.
Positive trials had a high likelihood to be non-inferior by their design (OR 3.8, 95% CI, 1.8-9.1, p = 0.001). This observation can be caused by the fact that margins in noninferiority trials are often chosen without clearly explained reasons, together with an asymmetry in how the trials are interpreted. The non-inferiority margin is a pre-specified amount in effect for which the investigated intervention is allowed to be worse than the comparator to still be called non-inferior. If the margin is prespecified to be larger than it should be it becomes harder for the investigated intervention to fall outside of this margin and be called inferior. Unfortunately, some studies show that 58% of non-inferiority margins provide no reason as to why they are selected [21]. Also, some studies suggest inconsistency in the CONSORT recommendation for the interpretation of non-inferiority trials [22].
The limitations of this study should be addressed. We have only analysed high quality evidence research from the CV setting, which was published in high impact medical journals in 2017, 2018 and 2019. Therefore, a larger observation period and the inclusion of a larger sample of journals and research with lower quality by their design could possibly render different results. Only primary manuscripts of randomized trials were included in our analysis. Since the interpretation of the trial results is made through the effect of intervention on the primary outcome, we have not included secondary manuscripts, such as reported subanalyses or follow up analyses. However, it should be emphasized that secondary analyses often get a lot of media attention which would have possibly affected the results of our study in case we included them in our analysis. Additional limitations of our paper could lie in the fact that the citation data could be incomplete, since we only used the WoS database for citation count which can provide a different profile of citation compared to other scholarly searched engines such as Google scholar [23]. Although this could affect absolute citation counts, it's effect on the relative citation effects between the positive and negative trials should be irrelevant. Although we have not performed power analysis, similar studies have used same time periods so we have speculated that this period should provide us enough trials to give us enough power to detect significant results. Despite some borderline results in terms of statistical significance that we have noticed in our analysis, we have not added more studies since this was not prespecified in our protocol. To our knowledge, this is the first paper which aimed to assess and compare the difference in impact of positive and negative CV randomized trials, measured through number of citations and AAS.

Conclusions
We did not find any difference between the impact of positive and negative CV RCTs considering number of citations. The higher number of published positive CV RCTs could raise suspicion on presence of publication bias associated with statistical significance, but further investigation of unpublished CV RCTs in trial registries (ClinicalTrials.gov) is needed to evaluate if this suspicion is true or false. Additional research is also necessary in order to shed more light on the role of using social media in the dissemination of CV research results and increasing it's scientific impact.

Ethics approval and consent to participate
Not applicable.