Leonard Freedman, GBSI JUDY LICHT, GSBI Leonard Freedman is on a mission to make science better. He’s one of the scientists who helped calculate a now-notorious statistic about waste in preclinical research—namely, that $28 billion in research funds is spent each year on irreproducible research, due to issues with biological reagents, laboratory errors, ill-planned experiments.
To help solve the “reproducibility crisis,” Freedman’s latest ambition is to train students in the fundamental principles of experimental design. The organization he leads, the Global Biological Standards Institute, recently received a $2.34 million grant to launch the project, entitled Producing Reproducible Experiments by Promoting Reverse Experimental Design. He spoke with The Scientist about where things are going wrong in experimental setups, and how his project will help solve the problem.
The Scientist: Where in the scientific process are things going wrong when it comes to irreproducibility?
Leonard Freedman: It’s multifaceted, there isn’t going...
That’s one of the reasons we’ve been focused on solutions around reagents and reference materials, but also we really thought hard about training, and in the context of training going back and teaching students and postdocs, and even early PIs, the importance of careful experimental design.
I would argue that together with reagents or the misuse or lack of authentication of experimental reagents, the poor way we do data analysis and reporting, study design, is as big an issue as any of those.
TS: Are there areas of life science, or types of experiments, that you think are hardest hit by the reproducibility issues?
LF: More in vitro types of experiments that involve cell lines and antibody reagents—which we know are problematic—might be more susceptible to issues around irreproducibility. But . . . I wouldn’t generalize in that way, because there are tremendous issues around using mouse models as well.
I don’t think . . . you could point to any one particular thing within the context of basic biological research and say that this area is worse than another area. Not to mention, something we haven’t really studied, but I’ve been thinking more and more about . . . the kind of incentives that people are being driven by. Like, I got to get this paper published, because I’ve got to get this grant. And it’s gotten more and more intense and competitive, and, you know, scientists are human beings like everyone else, and they get driven by these kind of, people call them perverse incentives. . . . I think that that’s a huge issue.
TS: How big of a driver is experimental design when it comes to irreproducibility in research?
LF: I would argue that together with reagents or the misuse or lack of authentication of experimental reagents, the poor way we do data analysis and reporting, study design, is as big an issue as any of those.
We’ve been working on some tangible solutions towards issues with reagents, like cell lines and antibodies. I think it when comes to something like study design, experimental design, it’s really about training, it’s really about going back to first principles and teaching, maybe in a more formal sense, teaching students, teaching postdocs, you know, the key fundamentals around how to design a good experiment. That was the real impetus for this grant that we received. With the expectation that if people are better prepared to design experiments, then they’re going to carry out the experiments, and almost by definition, that should improve the reproducibility of a given study.
See “The Rules of Replication”
TS: Where is current training, or teaching, in experimental design going wrong at the moment?
LF: This is my own personal perspective: I think graduate students and postdocs are taught at an apprenticeship kind of level. Courses are more formal, sort of theoretical, teaching about, you know, biological concepts, chemical concepts, or whatever. And then you get into a lab, and you start, and at that point it’s more of an apprenticeship relationship with your mentor. . . . And is there a formalized, didactic kind of training around fundamental things? Around study design or data analysis and reporting? Or the importance of validating your reagents? I think not.
What we’re suggesting is that we go back to basics, if you will, and really go back to these fundamental principles in terms of training students and postdocs. And if we do, I think that we will improve the quality of research.
Is there a formalized, didactic kind of training around fundamental things? Around study design or data analysis and reporting? Or the importance of validating your reagents? I think not.
TS: You want to address these issues by taking what you call a “reverse engineering” approach to experimental design. Can you explain how that solves part of the problem?
LF: In a nutshell, the idea around a reverse experimental design is that we’re beginning with the product. We’re beginning with essentially the data that is going to be generated using specific experimental methods. And in this particular grant we’re focusing on cell imaging, high throughput screening, and disease modeling. . . . Rather than starting at the beginning, talking more in abstract terms, you’re taking something tangible, you’re taking the product of the experiment, and working backwards to teach using very, very specific examples that are in front of the students, to teach the concepts, rather than . . . this being completely abstract.
See “The Great Big Clean-Up”
TS: As a whole, how do you expect the project to help irreproducibility in research?
LF: I think the real impact of our work and other people’s work in the reproducibility space could really be with . . . the future generation of scientists that are coming up and really are still being trained.
I think the real opportunity lies with better training, with an emphasis on fundamental issues like reagent validation, study design, and data analysis. These are things we all took for granted but maybe we shouldn’t be.
Editor’s note: The interview was edited for brevity.
Interested in reading more?
The Scientist ARCHIVES
Become a Member of
