multiple studies have found that the current peer-review system is unable to reliably identify the most promising research proposals. These findings add weight to the widespread sense among scientists, even successful ones, that the peer-review system is badly flawed.
Two studies, …clarify what peer review can and can’t do. In order to help the NIH select the best science, peer review must accomplish two things. First, it needs to weed out proposals that are seriously flawed or unimportant. And second, it needs to sort out the best research proposals from the merely good ones, because the budget isn’t large enough to fund all good proposals. The two recent studies of peer review show that peer review succeeds at the first task but not at the second.
When the researchers focused only on projects with competitive scores in the top 20th percentile or better, there was virtually no relationship between the peer-review score and the subsequent productivity of the research project. The only exceptions were projects with the very best scores, in the top 2nd percentile or above. These projects tended to produce more papers and receive more citations than projects with lower scores.
These results show that the NIH’s peer-review panels are, in fact, good at weeding out weaker proposals, and at picking a few obvious winners. But the problem is that most funding decisions happen between these two categories, where, the results show, peer-review scores are essentially useless.
Today, the NIH generally only funds grant proposals with scores above the top 15th or even 10th percentile…a funded project at the 8th percentile is not likely to be any better than an unfunded (and possibly never completed) project at the 18th percentile…