Let’s Fund: Better Science

Vaccines, modern agriculture, computers, evidence-based policies – scientific research produces enormous benefits to society.

But from climate science to cancer biology, science is facing a replication crisis. Lots of research findings can’t be repeated and billions of valuable research funding dollars might be wasted in the process.

Here’s a short explainer video about the replication crisis (for more in-depth analysis, see below):

 

There’s a new and better way of doing and publishing research, called Registered Reports, where research is peer-reviewed before the results are known. We’re crowdfunding for Professor Chris Chambers to work on getting scientific journals to adopt Registered Reports.

  • $38,234 raised of $75,000 51% 51%

Your donation is processed by GoFundMe and goes directly to Professor Chris Chambers at Cardiff University. $75,000 would free him of teaching and administrative duties at Cardiff University for one year.

Project summary

While not a ‘panacea’, Registered Reports might:

  • make science more theory-driven, open and transparent
  • find methodological weaknesses prior to publication
  • get more papers published that fail to confirm the original hypothesis
  • increase the credibility of non-randomized natural experiments using observational data

Currently, science is only peer-reviewed after the results are known. In contrast, in the Registered Reports format the peer reviewers accept scientific papers based on the methodology:

registeredreports_graphic

This format is applicable to a lot of areas in science and might help with the replication crisis.

Our first crowdfunding campaign, Let’s Fund #1, is raising funds for Professor Chris Chambers, the leading proponent of Registered Reports. This will free up his time to focus on accelerating the widespread adoption of Registered Reports by leading scientific journals.

This ‘meta-research’ project might be exceptionally high-impact because we can cause a paradigm shift in scientific culture, where Registered Reports become the gold standard for hypothesis-driven science.

The Grantee

Professor Chris Chambers

Professor Chris Chambers

Professor at Cardiff University

This campaign is raising funds for Professor Chris Chambers, the leading proponent of Registered Reports. Chambers is a Professor at Cardiff University. $75,000 would free him of teaching and administrative duties at Cardiff University for one year. Your donation will enable him to spend 50% of his time to focus on advocacy to accelerate the widespread adoption of Registered Reports by leading scientific journals.

Why should you donate to this campaign?

Scientific research produces enormous benefits to society. Developing medicines and vaccines, modern agriculture, computers, and evidence-based policies based on research that tries to find which ones work and what works best – all of these rely on the scientific method and have saved and improved millions of lives.

Read our full analysis on why research funding might be very high value.

Unfortunately, a lot of research findings are false and science is now facing a “replication crisis”. When repeating experiments, researchers from many areas of science are often unable to get the same results – with potentially billions of dollars of research funds being wasted in the process. We believe rather than donating to say cancer research by itself, it’s better to donate to ‘meta-research’, which tries to improve many areas of science at once, from economics to cancer biology.

Read our full analysis of meta-research and the replication crisis

One particularly promising meta-research project is the new Registered Reports publication format. It’s a new way of doing science that’s trying to help fix the replication crisis. Instead of scientists doing all the research and then submitting it for publication, they first write up the theoretical motivation and exact experiment and analysis they’re planning to run, submit that to the journal, and the research is approved if the method is sound – without the results.

Only then are the experiments run, and the results submitted. This leads to more theory-driven research, leaves less room for fishing for findings from data that didn’t fit the original hypothesis, and means researchers have an incentive to publish research even when nothing was found (and you would want to know if scientist found that a medicine didn’t work!).

Registered Reports has the potential to improve the scientific method – and so increase the bounty that scientific research produces.

Read our full analysis of the benefits of Registered Reports.

 

 

Figure taken from “First analysis of ‘pre-registered’ studies shows sharp rise in null findings | Nature” Oct. 2018

We want to fund Professor Chris Chambers at Cardiff University, the co-inventor and a leading advocate for Registered Reports, to work on getting more scientific journals to adopt this new standard. A buyout from his teaching would free up his time and our cost-effectiveness analysis suggests that he can advocate for journals to adopt the new format for between $365 and $1460 per journal.

Read our full analysis on the cost-effectiveness analysis of Registered Reports advocacy.

The number of journals adopting Registered Reports has been roughly doubling every year. So far, 140 journals have already introduced the Registered Reports format:

 

 
Study preregistrations on the Open Science Framework (OSF) are doubling every year; more than 140 journals have introduced registered reports. Figure taken from: “A recipe for rigor | Science.” Sep. 2018

Even though this is a good start, there are more than 25,000 scientific journals, and there is still a lot of room for advocacy.

But if enough reputable journals adopt this new format, then this might cause a paradigm shift in scientific culture, where Registered Reports would be seen as the gold standard for hypothesis-driven science. This would make whole fields, ranging from climate science to cancer biology to economics, more theory-driven, reliable and robust.

 

Our in-depth analysis

Why fund research? In general, funding research can be very effective in terms of expected value. Research has enabled incredible technological achievements; is quite tractable and ‘shovel-ready’ given that researchers are often looking for money to mobilize idle talent and can use existing university infrastructure; and both philanthropic and government spending on research should probably be higher given that richer countries tend to spend more on research.

Read our full report on why funding research is a good idea.

Why fund Meta-Research? Meta-research aims to identify and improve problems that exist across scientific disciplines (e.g. physics, biology, economics etc.) – for example, by improving common statistical methods. Surprisingly, our analysis shows that many hard sciences such as climate physics, macroeconomics, and cancer biology, have problems with reproducibility that meta-research addresses. By improving many fields of science at once, meta-research is a potentially high-value funding opportunity. Thus, donors might want to focus on indirect “superficial” contributions with wide applicability rather than relatively “profound” contributions limited to one narrow field, because funding meta-research can be much more effective than funding a single science “just as a lake can contain a lot more water than a well, even if the well is deeper”. Or as John Ionides puts it “Even 1% improvement in the yield and translation of useful discoveries effected through better research practices reflects value equivalent of many Nobel or Breakthrough prizes.”

Read our full report on the replication crisis and why funding meta-research might be.

Why fund Registered Reports advocacy? Registered Reports are a new way of doing and publishing research, that might lead to a quite radical change in scientific practice. The new format is a partial solution to the replication crisis in science. How do Registered Reports work? Initially, only the theoretical motivation for study and a very precise methodology are submitted for peer review to the journal– before any data is collected or analysis is performed. The paper is then accepted, rejected or accepted if revisions to the methodology are made. Only then, the data is collected and the analyses are performed. The paper will be published regardless of the outcome e.g. whether a drug worked or not. Here we argue that this approach is superior to traditional publishing. A more widespread adoption of Registered Reports would change scientific practice for the better through increased replicability of studies across many scientific disciplines, more theory-driven science, and potentially simplified grant-making. Advocacy for Registered Reports is relatively neglected by other funders and there’s unique window of opportunity to advocate for wider adoption of the format by more scientific journals. Despite some push-back from vested interests in the publishing industry, we believe that advocacy is tractable, and there is a chance that with a little push through increased funding, Registered Reports might become the gold standard for hypothesis-driven science.

Read our full report on why Registered Reports advocacy might make science more theory-driven, robust and replicable.

Why fund Chris Chambers? Professor Chris Chambers is a tenured Professor at Cardiff University, UK, and the co-inventor of the Registered Reports formats. He is already doing some Registered Reports advocacy but is time-constrained due to his university teaching load. Buying him out of his teaching and administrative work at the University would free up his time, so he could readily do more advocacy and introduce the Registered Reports to more and higher prestige journals. Our analysis suggests that funding this advocacy effort will lead to a cost-effectiveness of roughly $365-1460 per journal adopting the Registered Reports format, though this should not be taken literally. If a few hundred high-quality journals would sign up for the format, we believe it is not inconceivable that this could lead to non-linear increases in the adoption of the format and cause a paradigm shift in scientific culture, where the Registered Reports would be seen as the gold standard for hypothesis-driven science. We believe this is where most of the value of this project lies. We believe that this project has at the very least £112,000 ($150,000) room for more funding to free Chambers from teaching administrative duties for two years, but that the project could productively absorb even more so that funding is covered for several years. We close with a section in which we summarize some risks, reservations, and drawbacks of funding this grant.

Read our full report on the Cost-effectiveness of Registered Reports advocacy.

How does Registered Reports work?

The Center for Open Science describes how the Registered Report format works in detail:

Authors of Registered Reports initially submit a Stage 1 manuscript that includes an Introduction, Methods, and the results of any pilot experiments that motivate the research proposal. Following assessment of the protocol by editors and reviewers, the manuscript can then be offered in principle acceptance, which means that the journal virtually guarantees publication if the authors conduct the experiment in accordance with their approved protocol.

With this in hand, the researchers then implement the experiment. Following data collection, they resubmit a Stage 2 manuscript that includes the Introduction and Methods from the original submission plus the Results and Discussion. The Results section includes the outcome of the pre-registered analyses together with any additional unregistered analyses in a separate section titled “Exploratory Analyses”.

Authors must also share their data on a public and freely accessible archive … and are encouraged to share data analysis scripts. The final article is published only after this process is complete. A published Registered Report will thus appear very similar to a standard research report but will give readers confidence that the hypotheses and main analyses are free of questionable research practices.

Watch Chris Chambers give a talk about Registered Reports

“Even a 1% improvement in the yield and translation of useful discoveries effected through better research practices reflects value equivalent of many Nobel or Breakthrough prizes.”

Professor John Ioannidis

Professor, Stanford University

“As an academic with some experience in registered reports and a good view on the open science movement, I empathically think you should fund Prof. Chambers. I cannot think of a more efficient way to spend your money if you want to contribute to better science at this moment.”

Professor Tom Beckers

Professor, KU Leuven

“With Registered Reports, there is no scope for publication bias favouring positive findings (because the decision is made before the results are known), nor for the kind of analytic flexibility (p-hacking) that plagues many areas of science and leads to findings that fail to replicate.”

Professor Dorothy Bishop

Wellcome Trust Principal Research Fellow, University of Oxford

“We live in an age of information, but the quality of information is variable. Science is not immune to the growing suspicion that information that is provided may be selective or that there may be unintended (or even intended) biases in what is available. Scientists are increasingly seen as working toward a personal agenda rather than working towards the public good.

Registered Reports are an outstanding way to provide transparency.”

Professor Allen Moore

Editor-in-Chief, Ecology and Evolution

“Registered Reports should result in greater inclusiveness of researchers as they provide the opportunity to receive feedback on research design before data collection. This aspect of Registered Reports can support the development of more valid research designs by individuals in resource-poor positions, where access to feedback from colleagues and from conference attendance may be limited. Improved research designs would enhance these researchers’ ability to publish in high-visibility journals.”

Kaylin Ratner

Ph.D. Student, Cornell University

Why donate to this campaign? Our in-depth analysis