Introduction

Coronavirus Disease 2019 (COVID-19) pandemic is probably the most important challenge that most health care providers have or will have faced during their professional careers [1,2]. Not even experts were fully prepared to predict the extent of the spread of the disease [3,4] and its implications in our lifestyle and our way of practicing medicine [5,6,7]. Many specialists -and, in some countries, medical students or retired physicians- had to go back to the medical basics to give support to hospital units which have completely redefined their role, turning in most cases in monographic centers devoted to patients suffering from COVID-19 [7,8].

However, the commitment of the society and the restless work of all caregivers is not being enough to contain the rapid loss of human lives due to the complications of the disease. In fact, in certain regions, the high net reproduction rate (R0) [9] and the insufficient resources have limited the access to Intensive Care Units (ICUs), which, in some cases, has started to be regulated according to the principles of triage in a mass casualty scenario [5,7,10].

In this setting, looking for new treatments [11,12,13,14] becomes crucial. Many research groups have promptly reacted and started to carry out different studies in order to identify therapeutic targets or drugs with the objective of finding an adequate strategy to relieve the effect that this virus is producing worldwide. In addition, all scientific minds have turned to the interpretation of the data that the real-time experience facing this challenge was producing. In a setting of cold and darkness, any attempt to lay a fire is always welcome. However, the rush to obtain information and apply it in current clinical practice may lead to unpondered decisions that could produce more harm than benefit to the patients.

Following this, we aimed to evaluate the type of articles and the rate at which the evidence regarding this novel infection was becoming available.

Methods

We performed a non-systematic review of the literature including all the results retrieved between December 2019 and March 31^st, 2020, from the US National Library of Medicine database (Pubmed) after searching for the term ‘COVID-19’, without any other exclusion criteria, with the objective of showing the results retrieved by the simplest of the queries. Using the save function from the website, we downloaded a file including Pubmed identification number (PMID), title, authors, journal, publication date and Digital Object Identifier (DOI). The results were compiled in a Microsoft Excel (Redmond, WA) worksheet and subsequently imported in Stata 13.1 (StataCorp, TX), where all analyses were carried out.

Data was described using daily and weekly frequencies, medians, and ranges, including a graphical representation of the number of daily publications. Articles were also classified according to the quartile position of the journal where they were published, following the 2018 impact factor classification provided by the Journal of Citation Reports (JCR) from Clarivate Analytics (Philadelphia, PA).

With the objective of classifying the articles according to their type, we sampled the results to obtain a representative group of publications to be deeply analyzed. As the number of papers was known after the search (1901) -and conservatively estimating that 50% of the articles were review articles- we calculated (assuming that the proportions were following a normal distribution) that 320 papers (16.8%) had to be read to obtain proportions for each category along with their 95% confidence intervals. The articles were selected using a computer-generated randomization list.

For the selected publications, the title, the abstract and the full text were screened to identify the type of publication that best fitted in one of the following categories: reviews and interpretation of previously published data, case reports, mathematical and epidemiological modeling, reports of local experience, diagnostic studies, preclinical studies, experimental studies, and others. Further description of these categories can be found in Appendix 1. Classification was performed by two of the authors (FM and MA) and discordances were solved by discussion and further agreement. Then, following Wald’s normal approximation, we calculated the confidence intervals of these proportions.

Due to the nature of this study, institutional review board approval was not considered necessary.

Results

Between December 2019 and March 31st, 2020, 1901 articles were identified. The first report was published on December 13^th, 2019, therefore the search period comprised 110 days. As it can be seen on Figure 1a, the distribution of frequencies is asymmetrical, with a median of 7 publications per day (range 0-110). In fact, this rate accelerates over the ongoing days (Figure 1b), with a median of 11 (range 5-22) articles per day in the first week of the study period, which increased up to 78 (range 52-110) daily publications in the last week. Since January 28th, the mean addition to the previous daily number of releases was of 0.81 papers. Table 1 summarizes this data grouped by weeks.

The most common codes were T14.8XXA – fracture (185), M41.9 – scoliosis, unspecified scoliosis type, unspecified spinal region (42), R52 – pain (40), M41.125 – adolescent idiopathic scoliosis of thoracolumbar region (39), Q65.89 – developmental dysplasia of the hip (39), Z13.828 – scoliosis concern (35), Z98.890 – status post ORIF fracture (29), M54.5 – low back pain, unspecified back pain laterality, unspecified whether sciatica present (24), M79.672 – left foot pain (24), and M79.671 – right foot pain (21) (Table 1). The non-specific fracture code (T14.8XXA) was recorded in 16.84% of all encounters, and 81.52% of these fracture encounters had no other related ICD-10 codes to describe the fracture. Of all fractures coded, forearm fractures were the most common location of injury (33%, Figure 1). Significant discordance existed between providers regarding coding for fractures, as one provider used the non-specific code 54% of the time, which was significantly higher than the other two providers (38% and 41%, p=0.005 and p=0.042).

Regarding the journals where the articles were published, 1135 (59.7%) of them were situated in 1st quartile publications, while 313 (16.5%), 121 (6.4%) and 30 (1.6%) were in the 2nd, 3rd and 4th quartiles, respectively. Three hundred two (15.9%) works were distributed in journals which are not registered in the JCR.

Table 2 shows the proportion of each article type according to the abovementioned categories. Only three papers required in-deep reading by two authors to adequately classify them. As it can be seen, 57.5% of the papers were considered as interpretative reviews, while only 13.5% were case reports or reports of local experience and less than 0.5% were experimental studies.

Discussion

The increasing necessity to find answers to many of the questions that clinicians were daily facing at the saturated hospitals and overcrowded ICUs has inevitably led to the publication of many papers trying to achieve this goal. Searching the term ‘COVID-19’ on PubMed retrieves 1900 results in less than 4 months. This represented, up to that moment, more than 17 papers accepted and made available each day. This overwhelming amount of literature is impossible to read and interpret by first-line physicians, who are restlessly struggling against the disease in 12- or 24-hour shifts, and who must trust guidelines which, in the light of all this evidence, change in weekly timeframes. But even for policymakers -whose conclusions will decide the fate of many people in a very short period of time and who will have to be held accountable for their reactivity-, handling all these articles from a critical point of view, with an adequate analysis of the possible biases or alternative explanations to the findings, represents a titanic effort which may lead to misinterpretations that, in the best of the cases, will imply useless strategies, if secondary effects do not show up [15].

Therefore, the management of all the upcoming evidence should be relying on our reporting method, which is grounded on peer’s reviews and journal’s publication policies. It is completely understandable that COVID-19 has become the top-hit of publisher’s priorities and the efforts that they are doing by accelerating the editorial process to make evidence available as soon as possible. Nevertheless, peer-reviews (far from being perfect) have been conceived to ideally guarantee the quality of the research, favoring methods over results, and results over (mis) interpretations.

It is astonishing to find that more than 50% of these 1900 articles are opinions on clinical management, non-systematic reviews or comments on the data generated so far. Consistent evidence is scarce, considering that by that moment, almost nobody would have even thought of performing trials to test treatments against coronaviruses. Moreover, it is hard to believe that the process that goes from developing a hypothesis and conceiving the research protocol to its publication in a peer-reviewed journal (passing though institutional review boards, recruitment, collection of data, analysis, article writing, correction, submission, and review) could be achieved in only four months. And it is hard to imagine that physicians working in 12-hour shifts could barely have time to do so, then it should be others (who are not probably facing the human side of the disease) who will be fulfilling their resumes with 1^st-quartile publications.

The responsibility of journals in the adequate triage of papers becomes vital. The medical community has trusted them to communicate the results of the research to guide protocols and policies with direct effect on patient care, mainly due to their reputation regarding the strict scrutiny of the methodology of the papers they publish. It is still surprising that almost 60% of the abovementioned papers (which are far from being at the top levels of the hierarchy of the evidence quality) have been published in 1^st quartile journals. Therefore, it is worth asking if they are doing their best to provide evidence or if they are still stuck in the impact-factor race, more worried about losing their position in the JCR rather than providing the best available evidence, as most of them boast in their focus and scope section.

Following this, the question that may arise concerns the quality of the research that has been published so far. In this setting of uncontrolled spread of the disease, we would like to make available as much information as possible in the shortest of the timeframes. But in the middle of the rush of finding a solution to face the pandemic, the scientific community must stand steady and confidently in the defense of adequate research methodology [16]. We acknowledge the strata in quality evidence and we all know famous experiments that led to apparently good results which, when tested in general population, did not have the expected effect [17]. Hence, a big effort should be made to develop trials which will answer direct clinical questions, such as the “Solidarity” trial [18], and not to waste time in solitary fights and reports which, even if they are performed with the best of intentions, may present biases which may have an impact on our way of taking care of patients. We must etch in our minds that bad science kills people. Furthermore, in the middle of this catastrophe, the scenario is completely favorable to produce high quality information: we have the casuistic and a natural history of a disease that allows us to complete large clinical trials at a speed that no researcher would have ever imagined, with all the world (and all the sources of financing) rating this infection as a priority. Therefore, it is unacceptable that we lose this opportunity to perform top science and to find definitive answers to a problem that have become our everyday nature.

We owe the community the obligation of carrying out good science. Each patient who is registered to an uncontrolled trial is a waste of resources and, even worst, an irresponsibility towards a human being who has consented to be treated as an experimental subject. In fact, Ethics Committees should pay special attention to the protection of participants in the middle of this type of epidemies [19]. The hopelessness of becoming infected with a virus without treatment, while you are seeing your bedside partners being transferred to ICUs or dying, makes people vulnerable and ready to consent participation without thorough reflection. Review boards should always bear in mind this and not forget that, in human experimentation, global benefits do not justify individual pain, even in the situation of a pandemic.

Far from being a criticism to all those researchers who are doing their best to find answers in the middle of this chaos, we would like to pause for a while and think about what we are doing and where we are going. We are looking for solutions, but not every solution is valid. Bad science and unfounded recommendations may have serious implications (in terms of dangerous treatments, usual or infrequent secondary effects or unexpected risk consequences) [20], which will be multiplied by thousands considering the incidence of the disease. Hope is made visible when searching on NIH’s Clinicaltrials.gov, where more than 300 studies have been registered so far and their protocols are available for anyone willing to read them. We advocate for reducing as much as possible all the administrative procedures that may delay publication, but the development of the research should follow, now and always, the best of good clinical practices. We have the right to reject hypothesis or to have negative results, but methodology is unnegotiable.

Conflict of interest

The authors declare no conflict of interest.

Stand First

Migliorelli and colleagues state that today, more than ever, is the time to follow the principles of good science.

Key messages

• 1901 articles were published in 111 days, with a median of 11 publications per day during the first week, increasing to a median of 78 articles per day in the last week of the search period.
• It is impossible not only for first line caregivers, but also for policymakers, to analyze this tremendous amount of information and make the best decisions based on it.
• Journals should guarantee the strict scrutiny of the methodology of the papers they publish.
• The development of the research and the criteria for publication should follow the best of good clinical practices.

References

1. Smereka J, Szarpak L. COVID 19 a challenge for emergency medicine and every health care professional. Am J Emerg Med. 2020. DOI: 10.1016/j.ajem.2020.03.038.
2. Dehnavieh R, Kalavani K. Management-supportive measures for managers of healthcare organization during the COVID-19 epidemic. Infect Control Hosp Epidemiol. 2020;1-1. DOI: 10.1017/ice.2020.108.
3. Petropoulos F, Makridakis S. Forecasting the novel coronavirus COVID-19. PLoS ONE. 2020;15(3):e0231236. DOI: 10.1371/journal.pone.0231236.
4. Lipsitch M, Swerdlow DL. Defining the Epidemiology of Covid-19 – Studies needed. N Engl J Med. 2020;382:1194-6. DOI: 10.1056/NEJMp2002125.
5. Ferguson L, Barham D. Palliative Care Pandemic Pack: A Specialist Palliative Care Service response to planning the COVID-19 pandemic. J Pain Symptom Manage. 2020. DOI: 10.1016/j.jpainsymman.2020.03.026.
6. Simonato A, Giannarini G, Abrate A, Bartoletti R, Crestani A, De Nunzio C, et al. Pathways for urology patients during the COVID-19 pandemic. Minerva Urol Nefrol. DOI: 10.23736/S0393-2249.20.03861-8.
7. Ross SW, Lauer CW, Miles WS, Green JM, Christmas AB, May AK, et al. Maximizing the Calm Before the Storm: Tiered Surgical Response Plan for Novel Coronavirus (COVID-19). J Am Coll Surg. 2020, DOI: 10.1016/j.jamcollsurg.2020.03.019.
8. Carenzo L, Constantini E, Greco M, Barra FL, Rendiniello V, Mainetti M, et al. Hospital surge capacity in a tertiary emergency referral centre during the COVID-19 outbreak in Italy. Anaesth. 2020. DOI: 10.1111/anae.15072.
9. Viceconte G, Petrosillo N. COVID-19 R0: Magic number or conundrum? Infect Dis Rep. 2020;12(1):8516. DOI: 10.4081/idr.2020. 8516.
10. Scheidegger D, Fumeaux T, Hurst S, Salathé M. COVID-19 pandemic: triage for intensive-care treatment under resource scarcity. Swiss Med Wkly. 2020;150:w20229. DOI: 10.44147smw.2020.20229.
11. Sallard E, Lescure FX, Yazdanpanah Y, Mentre F, Peiffer-Smadja, N. Type 1 interferons as a potential treatment against COVID-19. Antiviral Res. 2020;178:104791. DOI: 10.1016/j.antiviral.2020.104791.
12. Cao B, Wang Y, Wen D, Liu W, Wang J, Fan G, et al. A Trial of Lopinavir–Ritonavir in Adults Hospitalized with Severe Covid-19. N Eng J Med. 2020; March 20: 1-13. DOI: 10.1056/NEJMoa2001282.
13. Russell B, Moss C, Rigg A, Van Hemelrijck M. COVID-19 and treatment with NSAIDs and corticosteroids: should we be limiting their use in the clinical setting? ecancer. 2020;14:1023. DOI: 10.3332/ecancer.2020.1023.
14. Insiat Islam Rabby MD. Current Drugs with Potential for Treatment of COVID-19: A Literature Review. J Phar Phar Sci. 2020;23(1):58-64. DOI: 10.18433/jpps31002
15. Correia MI, Ramos RF, Von Bahten LC. The surgeons and the COVID-19 pandemic. Rev Col Bras Cir. 2020; 47:e20202536. DOI: 10.1590/0100-6991e-20202536
16. Zhang T, He Y, Xu W, Ma A, Yang Y, Xu KF.Clinical trials for the treatment of Coronavirus disease 2019 (COVID-19): A rapid response to urgent need. Sci China Life Sci. DOI: 10.1007/s11427-020-1660-2.
17. Ioannidis, JPA. Why most published research findings are false. PLoS Med 2005;2(8):e124. DOI: 1371/journal.pmed.0020124.
18. “Solidarity” clinical trial for COVID-19 treatments. 2020. Available from: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov/solidarity-clinical-trial-for-covid-19-treatments.
19. Qiankun L, Tao Q. Managing clinical trials for covid-19: the importance of ethics committees. BMJ. 2020;369:m1369. DOI: 10.1136/bmj.m1369.
20. FitzGerald GA. Misguided drug advice for COVID-19. Science. 2020;367:1434. DOI: 1126/science.abb8034.

<