Advertisement

A new machine-learning tool proposes to comb through PubMed’s database in an effort to combat the “reproducibility crisis” across scientific disciplines.

The “R-factor” measuring tool, created by the Connecticut-based Verum Analytics, debuted last week.

The concept was originally unveiled by the company in a preprint research paper posted online last year

The machine assesses a paper based off the number of times it is cited by other studies – and then whether its claims are verified or unverified. The “R-factor” is thus determined between 0 (meaning completely unverified) and 1 (totally validated), the developers explained.

That value is calculated by dividing the number of reports that validate the claims by the number of attempts to do so. The citations that just mention the claim without testing do not factor into the formula, they said.

“The R-factor is also based on another principle of science, that new research should proceed from a comprehensive understanding of previous work,” they wrote in their introductory paper last year. “Following this principle is becoming more difficult because the sheer number of publications overwhelms even experts, thus making their expertise even more narrow.

“The R-factor would help to solve this problem not only by providing a measure of veracity for a published claim, but also by indicating the studies that verified or refuted it.”

The debut of the tool was first reported in Science. The magazine quoted several experts’ doubts on the utility of the tool – especially since publication bias may prompt journals to publish positive results over negative results. Another limitation the magazine cited is that the R-factor only accounts for papers publicly available in the PubMed database – and not behind a paywall.

Laboratory Equipment attempted to use the Verum Analytics machine to verify the R-factor of one of the most notorious papers in history: the retracted 1998 article by Andrew Wakefield and colleagues in The Lancet which first made connections between vaccines and autism. After 10 minutes, the site said it was still “searching for a robot” to work on assessing the paper.

The “reproducibility crisis” has only grown, according to a major study published in the journal Royal Society Open Science in September 2016. The pressure to “publish or perish” means that “low power” results including small effect sizes, false positives, and low reproducibility and replication rates have multiplied. One the most glaring examples in recent months was shown when a Chinese agency admitted that scientists in the country had engaged in massive peer-review fraud in cancer studies leading to 107 retractions from the Springer journal Tumor Biology.

Advertisement
Advertisement