I have been thinking a lot about reproducibility in science recently. Specifically, I remember one time when I told Peter Walter my research had been unable to replicate a finding. Peter told me to read his article, titled “On Reproducibility and Clocks”. I tried, but was unable to find it (the URL no longer seems to exist). Peter’s point was, I think, that reproducibility is hard, and often failed attempts at reproducibility often reflect that something changed between the original attempt and the replication attempt just as much as they reflect that the original finding is wrong.

Reproducibility is an uncomfortable spectrum. Science is a meandering art, and as a result (often) we arrive at a discovery or a conclusion by fine-tuning parameters, sometimes unwittingly. When we report our findings, we include a thorough description of all the conditions necessary to reproduce an experiment, but even the most experienced scientist may omit some conditions. These conditions are frequently taken for granted or may be unknown: they are latent.

As scientists replicate one another’s work, these latent conditions are made explicit through time. RNAi was known for a long time, but a necessary condition for RNAi to work (double strandedness) remained elusive until a brilliant set of experiments by Craig Mello and Andrew Fire clarified it. Thus, the cutting edge of science can be frequently wrought with irreproducible results that are only much later explained. In other words, reproducibility is as much a sign of reality as it is of the robustness and maturity of a protocol.

This has practical implications for startups. In biotechnology, we like to view startups as developing cutting-edge science and technology. Therefore, reproducibility may be lower for startups at early stages. Biotechs therefore must learn to cultivate a love of irreproducibility. Irreproducibility can be a positive signal that people are pushing boundaries.

That said, it also seems that biotech should fall in love with robustness. The first step when building a scientific team around a new finding has to be to democratize this discovery, to ensure that every scientist on the team can and does, to some degree or another, perform the same scientific protocols and arrive at similar if not identical conclusions. It strikes me that often as scientists we can be very defensive of sharing protocols, training others to do what we do, or recognizing that the way we design our experiments and protocols is not robust.

How should organizations best ensure that they phase the discoveries and technologies they are based on from cutting-edge and technically hard to reproduce towards robust and well-understood scientific and engineering knowledge? I would be curious to hear your thoughts.