To what extent is science a strong-link problem?

Here’s a fascinating and worrying news story in Science: a top US researcher apparently falsified a lot of images (at least) in papers that helped get experimental drugs on the market — papers that were published in top journals for years, and whose problems have only recently become apparent because of amateur sleuthing through PubPeer.

I’m going to wane philosophical for a minute. In general I’m very sympathetic to Adam Mastroianni’s line “don’t worry about the flood of crap that will result if we let everyone publish, publishing is already a flood of crap, but science is a strong-link problem so the good stuff rises to the top”. I certainly don’t think we need stronger pre-publication review or any more barrier guardians (although I have reluctantly concluded that having some is useful). But when fraudulent stuff like this does in fact rise to the top in what seems to be a strong-link network — lots of NIH-funded labs, papers in top journals (or, apparently, “top” journals) — then I despair a bit. Science has gotten so specialized that almost anyone could invent facts or data within their subfield that might pass muster even with close colleagues (even if those colleagues aren’t on the take, he said cynically — there is a mind-boggling amount of money floating around in the drug-development world).

Immediate thought experiment: could Mike or I come up with material for a blog post or paper that would be false but good enough to fool the other? Given how often we find surprising or even counterintuitive results, I think possibly so. I’m not particularly motivated to run the experiment when we’re already digging out from a deep backlog of started-but-never-finished papers, but it remains a morbidly fascinating possibility.

Anyway, one problem is that “top” journals have a lot of fraudulent or at least incorrect science in them, roughly corresponding to their impact factors. Now, you might say “yeah but the positive correlation means bad actors get caught”, to which I’d reply “not fast enough” and “how do you know we’re catching all of them?”

Sinking feeling

There’s another problem, I don’t know if it’s equal-and-opposite but it definitely exists: good science that doesn’t float to the top. Here are a couple of quick examples from my neck of the woods:

Working from very little evidence by modern standards, Longman (1933) had correctly figured out that pneumatic sauropod vertebrae come in two flavors, those with a few large chambers and those with many small chambers. He called them “phanerocamerate” and “cryptocamerillan”, corresponding to the independently-derived modern terms “camerate” for the open-chambered form and “camellate” or “somphospondylous” for the honeycombed one. As far as I have been able to determine, nobody paid any attention to this before Wedel (2003b) — Longman’s work on vertebral internal structure wasn’t mentioned or cited by Janensch in the 1940s or Britt or anyone else in the 1990s. To be clear, I’m not putting myself forward as a better researcher than anyone that came before. I just got lucky, to have read a fairly obscure paper while I had my antennae out for any possible mention of pneumaticity.

Speaking of Janensch, his 1947 paper on pneumaticity in dinosaurs was pretty much ignored until the 1990s and early 2000s.

I owe my career to the Dinosaur Renaissance

Here’s what bothers me about this: I made my career studying pneumaticity in sauropods, buoyed in large part by the fact that I stumbled backwards into a situation where I had access to a big collection of sauropod bones (at the OMNH), free time on a CT scanner (at the university hospital), and a curious and collaborative radiologist (Kent Sanders). But you don’t need a CT scanner to study pneumaticity, as John Fronimos has convincingly demonstrated (see Fronimos 2023 and this post). So why didn’t the revolution in sauropod pneumaticity happen in 1933 or 1947? Or, heck, in 1880 — Seeley and Cope and Marsh and many others recognized that sauropods had highly chambered vertebrae.

I think the most likely explanation is that at the time no-one cared. Pneumatic vertebrae in sauropods were possibly interesting trivia, but sauropods were an evolutionary dead end and so their vertebrae couldn’t tell us anything important about evolutionary success. These attitudes may not have been universal, but they were certainly prevailing.

I had the good fortune to come along at a time when there was renewed interest in dinosaur paleobiology, particularly any characters or body systems shared between non-avian dinosaurs and birds. Suddenly pneumaticity wasn’t some obscure bit of trivia, but the skeletal footprint of a bird-like respiratory system that was potentially a key adaptation for sauropods (Sander et al. 2011) and possibly for dinosaurs more generally (Schachner et al. 2009, 2011). And dinosaurs weren’t any more of an evolutionary dead end than we are, they just happened to mostly not fit into small holes or deep water when the asteroid hit.  (Let’s heat the atmosphere to 400F for a few hours and then make the world dark for a few months or years and then we can talk about evolutionary dead ends.) So adaptations that facilitated dinosaurosity might tell us something about evolutionary success after all.

What are you doing in that cell?

Having a successful career because I happened to hitch a ride on a wave of renewed interest in dinosaur paleobiology is certainly nice, but also worrisome. If it takes 70 or 100 years for the good science to float to the top, does that really count? Whatever convection cells push the good science toward the top would ideally work more like a cook pot on a rapid boil, and not like the imperceptible roiling of Earth’s mantle. So ask yourself: what’s still on its way up to the top right now, that no-one has clocked yet? What’s the Longman (1933) of 2024 — the seemingly incidental observation that is going to seem prophetic in a few decades? Or worse, what was the Longman (1933) of 1994 or 2004, the solid paper that attracted no attention and won’t for another half century?

The convection cell metaphor is particularly apt because a lot of science is siloed. A good idea — say, that the peroneus tertius muscle occurs at a lower frequency in monkeys and apes than in humans, and this tells us something about its evolution — may rise to the top in one cell (comparative anatomy), but not make it over to the neighboring cell (clinical anatomy), where all the happy little molecules think that peroneus tertius is a muscle unique to humans (if you have no idea what I’m on about, see the second numbered point in this post).

So if you want to do good work — in this metaphor, to be at the top where the good science floats (eventually, alongside a seasoning of not-yet-unmasked bad science) — then I think you have to be aware that other cells exist, and occasionally peer into them, if for no other reason than to make sure you don’t accept an idea that’s already been debunked over there. And you need to read broadly and deeply in your own cell — there’s almost certainly valuable stuff you don’t know because the relevant works are stuck to the bottom of the pot. Go knock ’em loose.

References

• Fang, F.C. and Casadevall, A. 2011. Retracted science and the retraction index. Infection and Immunity 79(10):3855-3859.
• Fronimos, John A. 2023. Patterns and function of pneumaticity in the vertebrae, ribs, and ilium of a titanosaur (Dinosauria, Sauropoda) from the Upper Cretaceous of Texas, Journal of Vertebrate Paleontology 43:2. DOI: 10.1080/02724634.2023.2259444
• Janensch, W. 1947. Pneumatizitat bei Wirbeln von Sauropoden und anderen Saurischien. Palaeontographica, Supplement 7, 3:1–25.
• Longman, H. A. 1933. A new dinosaur from the Queensland Cretaceous. Memoirs of the Queensland Museum 10:131–144.
• Sander, P.M., Christian, A., Clauss, M., Fechner, R., Gee, C.T., Griebeler, E.M., Gunga, H.C., Hummel, J., Mallison, H., Perry, S.F. and Preuschoft, H. 2011. Biology of the sauropod dinosaurs: the evolution of gigantism. Biological Reviews 86(1):117-155.
• Schachner, E.R., Lyson, T.R. and Dodson, P., 2009. Evolution of the respiratory system in nonavian theropods: evidence from rib and vertebral morphology. The Anatomical Record 292(9): 1501-1513.
• Schachner, E.R., Farmer, C.G., McDonald, A.T. and Dodson, P., 2011. Evolution of the dinosauriform respiratory apparatus: new evidence from the postcranial axial skeleton. The Anatomical Record 294(9): 1532-1547.

Source: https://svpow.com/2024/10/30/to-what-extent-is-science-a-strong-link-problem/