Between the Lab and the Library: The Risks in Trying to Co-opt 'Gold Standard Science'

A response to a high-profile proposal to enforce ‘gold standard scholarship’.

The Editor-in-Chief of Science Magazine, Holden Thorp, released an Editorial on Thursday 9/18 entitled “Gold standard science requires gold standard scholarship.” The central premise is to equate (accusations of sometimes sloppy) citation practice in scientific research with that of deliberate research misconduct. In doing so, the Editorial takes the administration’s banner of ‘gold standard science’ and proposes that stricter enforcement of ‘gold standard scholarship’ is required to fix perceived ills in the science ecosystem. I argue that the Editorial takes the administration’s bad-faith argument and runs with it – abetting an anti-science movement with policy recommendations that have the potential for misuse to the detriment of science.

Editorial by EIC of Science, 9/18.

The editorial’s timing is interesting on many levels. It was published amidst a week where the former CDC Director Susan Monarez testified about her firing before the Senate’s HELP committee and as the newly assembled ACIP committee including anti-vaccine skeptics & activists take steps to reduce access to childhood vaccines. None of this is consistent with any reasonable notion of ‘gold standard science’.

Precisely so, we should exercise far more caution before accepting a bad-faith argument about 'restoring gold standard science' from an administration that released an AI slop MAHA report, removed expert advisory committees at CDC and NIH, terminated projects at institutions throughout the US, and appointed someone who was sanctioned for practicing medicine without a license in Maryland with the lead role in fast-tracking a CDC study to identify the putative causes of autism.

Despite having all of these facts in hand, the framing of this editorial suggests a false equivalency between deliberate administration steps to fit evidence to predetermined policy objectives (i.e., definitely not science) with the known problem that science is imperfect and citations aren’t either (i.e., part of self-correcting science). I am not defending imperfect citations. Indeed, inappropriate citations are not the only issue; what happens when authors omit relevant citations because prior work does not support their narrative? Yes, some “inappropriate” citations may be deliberate (which should be called out by reviewers/editors and have consequences, as already unfolds in practice) while other citations may be flagged as “inappropriate” by the original authors of the cited work for a more banal reason: science is nuanced.

Let’s take one example.

Thorp’s editorial cites a study that finds 17% of citations “do not support the statement to which they are applied.” That is indeed a top-line conclusion of the study and one that Thorp uses to drive his argument. But it’s not the whole story. The same study had a 98% non-respondent rate, i.e.,. only 2% responded; 2,648 respondents of 127,928 invitations). In other words, most researchers ignored the request; this level of response could have led to biases. The paper authors were aware of this issue: “Indeed, in asking authors to assess citations of their own work we are asking the people perhaps most likely both to value the contribution of the cited work, and understand its nuances, and therefore potentially be more likely to be sensitized to what they perceive as quotation errors.” That seems like important context to have been shared in the editorial. Imagine if the study authors had received Thorp’s editorial: would this count as a citation error? I think Thorp is right to quote the 17% number, but it might be fair to say that the citation was an oversimplification – some of this is indeed in the eye of the beholder.

But the problem of citations, caveats and all, represents a very different matter than what the administration is doing while waving the banner of ‘gold standard science’.

Precisely given the bad faith arguments of the administration, we should not embark on that path that Thorp recommends, in which “misrepresentation of references is treated as seriously as research misconduct.” We already take lots of things seriously in science. There are real consequences in place for actual research misconduct. I am surprised the editorial did not at least include the phrase “deliberate misrepresentation”, given the potential for misuse (as addressed elsewhere).

An administration intent on selective enforcement could decide to take Thorp’s recommendations both seriously and literally, and utilize review of references and spurious charges of ‘misrepresentation’ as a means to punish perceived opponents. This misuse could apply in cases where the citing authors and the cited authors disagree on the key take-away message of a cited reference or perhaps because the administration’s appointees become bad-faith arbiters of such decisions.

The well-known problem of (imperfect) citation practice is already addressed by panels, journals, editors, reviewers, RCR classes and more, on top of the reputational consequences of writing sloppy papers that other people read. Is this system perfect? No and increasing safeguards is surely a good thing. But is current citation practice analogous to deliberate research misconduct or the real-world consequences of politicization and dismantling of science? No.

Science and society (and the ongoing dialogue between them) are about to face even bigger problems. Thorp’s editorial advises scientists to ‘calmly and methodically’ evaluate evidence when HHS releases its anticipated autism report that promised an answer on a deadline. The newly composed ACIP is meeting this week and already voted to restrict access to MMRV vaccines for younger children. There are real-world consequences to removing experts from these positions and replacing them with individuals who do do not follow an evidence-to-recommendation pathway (see Inside Medicine for more). Extraordinary claims require extraordinary evidence. Being calm and methodical is good, but scientists and health organizations will have to be assertive and prompt in responding whether to ACIP or the pending HHS report.

Perhaps the upcoming HHS report will have its citations in order, perhaps not. But, if the report reaches the conclusions that Sec. Kennedy and allies favor, then the report may have methods and data that are problematic or misused (or perhaps wholly unavailable). It is also possible that the report will cherry pick its studies. The issue of citation checking is likely secondary. The choice to take the framing of ‘gold standard science’ and champion it as the basis for ‘gold standard scholarship’ overlooks something far more consequential.

An administration whose health science leadership has cancelled billions of dollars of awarded grants, removed panelists, and insist that data should fit a predetermined narrative aren’t really practicing “gold standard science” no matter what the citation format is or how good the slogan sounds. To compare that to the real practice of scientific research by experts throughout the US accepts the bad-faith argument of the administration as a means to malign what has been (and still could be) a world-leading system of science, scientists, and institutions.

A few years ago, Thorp addressed what was at stake in a Science editorial: “The scientific community is up against a sophisticated, data-driven machine that is devoted to making sure that science doesn't fully succeed, and the history of this is quite clear.” He also had a substantive recommendation, far more pro-active than citation checking: “The only way to win this fight is to harness the same sophisticated tools in the name of science that are being used to tear science down.”

I agree.