The credibility of preprints can range from very high to very low, but there are ways to gauge their credibility, and their quality. These ways include looking at any available data associated with the manuscript, reviewer comments, the author’s reputation, and the preprint’s status in the peer review process–if the preprint server offers that feature. Preprint servers’ screening standards, quality controls, and other features can also contribute to these preprints’ overall quality. Quality and credibility are something all authors’ should aim for. Here are ways you can achieve them.
Ways of finding the credibility of a preprint
There are a number of ways to discern a preprint’s credibility. Asking yourself the following as you read them will help you decide.
What data are associated with the preprint’s scientific findings?
Some researchers will upload data associated with their preprints, if the preprint server allows it. This can include research data, materials, code, and other supplementary files. These files back up the figures, results, and other sections associated with the research and allow analyses to be reproduced.
Preprints already promote openness and transparency. But when authors share their data and other supplementary information in their preprints, it reinforces their credibility.
What is the scientific community saying about the preprint?
Some dedicated preprint platforms, as well as preprint servers like Research Square, allow comments. Commenters can be known colleagues in the field, researchers in tangential fields, or even enthusiasts. These “community reviews” can contribute important context and call out the strengths and weaknesses of a paper.
Robust feedback – even if that feedback includes disagreements – should not be viewed as a sign of a paper’s weakness. In fact, serious engagement is often a positive sign of the work’s credibility. Conversely, having no public feedback may not indicate a paper’s weakness; some scientists just prefer to share comments directly with authors via email.
Here’s an example of a comment from here on Research Square. This particular comment highlighted the importance of this planetary gravitational field study as a “new evaluation strategy for the prediction of earthquakes.”
What is the author’s reputation?
All preprint servers provide information about the authors. This can be used to further investigate the authors themselves.
Google Scholar, for example, can provide useful metrics to help uncover an author’s reputation, such as:
Also, what institutions are the authors and co-authors affiliated with? Do they work at a well-known institution, like Harvard or the Salk Institute for Biological Studies, or are they from a virtually unknown organization?
Keep in mind, however, that an author’s institution is not the best indicator of that researcher’s reputation. Great research can come from anywhere. Well-known researchers from highly regarded institutions have actually engaged in fraudulent research activities.
What is the preprint’s status in the peer review process?
A small but growing number of preprint platforms offer integrations with peer-reviewed journals, allowing readers to see their manuscript status during the peer-review process. service is one example of this.
Through In Review, authors can submit preprints to Research Square as they submit their manuscripts to one of 600+ participating journals.
After authors submit their articles, a peer review timeline showing the manuscript’s status – e.g., Under Review’, Under Revision, or ‘Published’ – is visible on the preprint.
“Published” status indicates the manuscript has passed peer review and has been published in the partner journal. This is also the most definitive indicator of a preprint’s credibility. It shows that the manuscript has made it through the peer review process.
Other ways of gauging credibility
Other information on the peer review timeline can help a reader gauge the preprint’s credibility. For example, did the peer reviewers prescribe major or minor editorial revisions to the original preprinted manuscript?
If the revisions were minor, it’s likely that the research is generally credible.
If major revisions were needed, it’s likely that at least the first version of the preprint may have had considerable issues or may have been missing important information. This credibility could increase in subsequent versions, however, after authors make those major revisions.
The first version of the following preprint was submitted on April 8, 2022. The editorial decision (“Major revision”) was logged into In Review just one week later. The status of the paper is currently Under Revision.
Does the preprint show an indicator of quality or certification?
Researchers may opt for pre-certification as a way to show the credibility of their manuscript, or the data and methodologies behind it, before submission to a journal. These are often certificates provided through various editorial services, such as editing and scientific reviews.
These quality indicators may be visible on the preprint itself.
Research Square, for example, offers two types of badges as an indicator of quality:
Here’s a preprint that earned the Methods Reporting badge.
Preprint servers and their role in preprint credibility
The credibility of preprints often starts with the quality of the preprint servers hosting them. Look at the server’s preprint screening processes, level of management, details of their editorial policies, and other standards. All can vary widely between servers. These are clues about the servers’ credibility. Here’s how to interpret them.
How comprehensive is the preprint server’s screening process?
Preprint servers’ staff may ask a rigorous series of questions before submitting a preprint posting.
Some preprint servers, such as those run by societies or proprietary organizations, might gate their content in other ways. For example:
For a comprehensive list of preprint servers and their screening processes, visit ASAP Bio’s Preprint Server Directory.
Is the preprint server actively managed?
A preprint server should have a team of individuals who ensure that submissions are properly screened and processed before they become preprint postings. Clear leadership, editorial staff, and an editorial board should guide these teams.
The server is a leading authority
Is there a visible editorial leader behind the preprint server? For example, is there an editor-in-chief, founder, or other other individual actively managing the content, the day-to-day operations of the preprint server, and perhaps serving as a thought leader in the preprint sphere? These would lend credibility to the preprint server.
Editor in Chief Michele Avissar-Whiting is the leading editorial authority at Research Square. She actively manages a team of experts who screen preprints. Dr. Avissar-Whiting also leads the Research Square Advisory Board and is a thought leader in the preprint space, authoring both original research and thought leadership articles on the topic.
The server has a dedicated editorial staff
Preprints are not peer-reviewed, but they should be screened by human eyes before posting. Even more importantly, these humans should have some expertise with research. Therefore, another indicator of credible preprints is a staff of dedicated reviewers screening them against editorial policies and preprint screening standards.
Also important as you gauge the credibility of preprints and preprint servers:
At Research Square, for example, five editorial staff members with advanced science degrees are dedicated to screening roughly 1,500 preprints each week.
Many preprint servers advertise their practices. You’ll often find such information on each preprint server’s website.
The server has an editorial or advisory board
Editorial and advisory boards are also a good sign of a credible preprint server. These board members can include leading researchers, librarians, publishing industry experts, and other key stakeholders.
Such experts can offer academic guidance on the platform and ensure that it meets the needs of the targeted research community.
Conclusion on preprint and preprint server credibility
Preprints posted to preprint servers are not peer reviewed like manuscript submissions to traditional scientific journals, but the majority of them are indeed credible.
Most submitted preprints go through editorial checks and standards for quality. Many preprints through community reviews that help improve them; and the majority of preprints pass the peer-review process and go on to publication in journals.
Submitting a research manuscript soon? Research Square, offers a variety of presubmission and post-publication services to ensure your research is of the highest quality before journal submission.
References
“Systematic examination of preprint platforms for use in the medical and biomedical sciences setting”, Kirkham et al., BMJ Open. 2020. http://dx.doi.org/10.1136/bmjopen-2020-041849
“Credibility of preprints: an interdisciplinary survey of researchers” Courtney K Soderberg, et al. Royal Society Open Science. 28 October 2020.
“Meta-Research: Tracking the popularity and outcomes of all bioRxiv preprints”, Richard J Abdill and Ran Blekhman, eLife. 24 April 2019.
https://elifesciences.org/articles/45133
