Let’s Fund is a crowdfunding site that helps you discover, learn about and fund breakthrough research and policy projects.
We help you make an informed donation to high-risk, high-reward projects: we give you in-depth analysis using the principles of Effective Altruism, which lets you maximize the social impact of your giving.
Let’s Fund #1: Better Science
Vaccines, modern agriculture, computers, evidence-based policies – scientific research produces enormous benefits to society.
But from climate science to cancer biology, science is facing a replication crisis. Lots of research findings can’t be repeated and billions of valuable research funding dollars might be wasted in the process.
Here’s a short 4-minute explainer video about the replication crisis from TED.Ed (you can skip to more in-depth analysis below):
So what, concretely, can we do about this?
There might be a new and better way of doing and publishing research, called ‘Registered Reports’, where research is peer-reviewed before the results are known.
While not a ‘panacea’, Registered Reports might:
- make science more theory-driven, open and transparent
- find methodological weaknesses prior to publication
- get more papers published that fail to confirm the original hypothesis
- increase the credibility of non-randomized natural experiments using observational data
Currently, science is only peer-reviewed after the results are known. In contrast, in the Registered Reports format the peer reviewers accept scientific papers based on the methodology:
This format is applicable to a lot of areas in science and might help with the replication crisis.
Our first crowdfunding campaign, Let’s Fund #1, is raising funds for Professor Chris Chambers, the leading proponent of Registered Reports. This will free up his time to focus on accelerating the widespread adoption of Registered Reports by leading scientific journals.
This ‘meta-research’ project might be exceptionally high-impact because we can cause a paradigm shift in scientific culture, where Registered Reports become the gold standard for hypothesis-driven science.
Read on if you like to read our in-depth research on this topic.
“Even 1% improvement in the yield and translation of useful discoveries effected through better research practices reflects value equivalent of many Nobel or Breakthrough prizes.” John Ioannidis, Professor at Stanford University
“As an academic with some experience in registered reports and a good view on the open science movement, I empathically think you should fund Prof. Chambers. He has been the single most important driving force in implementing the format of registered reports at an increasing range of psychology journals. He is clearly someone who can make change happen; his impact over the past few years has been enormous. Problems of replicability are by no means limited/specific to psychology, however. Investing in the widespread adoption of registered reports across the empirical sciences is surely one of the most efficient ways to improve the reliability of scientific discoveries. And funding Prof. Chambers is probably the most efficient way to invest in the further development and adoption of the registered report format. I cannot think of a more efficient way to spend your money if you want to contribute to better science at this moment.” Tom Beckers, Professor at KU Leuven
Currently raised: $5 of $75,000
Your donation goes to Professor Chambers
proccessed via GoFundMe and
Meet the grantee
Our in-depth analysis
The executive summary: why should you donate to this campaign?
Scientific research produces enormous benefits to society. Developing medicines and vaccines, modern agriculture, computers, and evidence-based policies based on research that tries to find which ones work and what works best – all of these rely on the scientific method and have saved and improved millions of lives (read our analysis on why research funding is a great idea).
Unfortunately, a lot of research findings are false and science is now facing a “replication crisis”. When repeating experiments, researchers from many areas of science are often unable to get the same results – with potentially billions of dollars of research funds being wasted in the process. We believe rather than donating to say cancer research by itself, it’s better to donate to ‘meta-research’, which tries to improve many areas of science at once, from economics to cancer biology (read our analysis of meta-research and the replication crisis).
One particularly promising meta-research project is the new Registered Reports publication format. It’s a new way of doing science that’s trying to help fix the replication crisis. Instead of scientists doing all the research and then submitting it for publication, they first write up the theoretical motivation and exact experiment and analysis they’re planning to run, submit that to the journal, and the research is approved if the method is sound – without the results.
Only then are the experiments run, and the results submitted. This leads to more theory-driven research, leaves less room for fishing for findings from data that didn’t fit the original hypothesis, and means researchers have an incentive to publish research even when nothing was found (and you would want to know if scientist found that a medicine didn’t work!).
Registered Reports has the potential to improve the scientific method – and so increase the bounty that scientific research produces (read our analysis of the benefits of Registered Reports).
We want to fund Professor Chris Chambers at Cardiff University, the co-inventor and a leading advocate for Registered Reports, to work on getting more scientific journals to adopt this new standard. A buyout from his teaching would free up his time and our cost-effectiveness analysis suggests that he can advocate for journals to adopt the new format for between $365 and $1460 per journal (read our cost-effectiveness analysis of Registered Reports advocacy).
The number of journals adopting Registered Reports has been roughly doubling every year. So far, 140 journals have already introduced the Registered Reports format:
Even though this is a good start, there are more than 25,000 scientific journals, and there is still a lot of room for advocacy.
But if enough reputable journals adopt this new format, then this might cause a paradigm shift in scientific culture, where Registered Reports would be seen as the gold standard for hypothesis-driven science. This would make whole fields, ranging from climate science to cancer biology to economics, more theory-driven, reliable and robust.
Our in-depth analysis
Why fund research? In general, funding research can be very effective in terms of expected value. Research has enabled incredible technological achievements; is quite tractable and ‘shovel-ready’ given that researchers are often looking for money to mobilize idle talent and can use existing university infrastructure; and both philanthropic and government spending on research should probably be higher given that richer countries tend to spend more on research.
Why fund Meta-Research? Meta-research aims to identify and improve problems that exist across scientific disciplines (e.g. physics, biology, economics etc.) – for example, by improving common statistical methods. Surprisingly, our analysis shows that many hard sciences such as climate physics, macroeconomics, and cancer biology, have problems with reproducibility that meta-research addresses. By improving many fields of science at once, meta-research is a potentially high-value funding opportunity. Thus, donors might want to focus on indirect “superficial” contributions with wide applicability rather than relatively “profound” contributions limited to one narrow field, because funding meta-research can be much more effective than funding a single science “just as a lake can contain a lot more water than a well, even if the well is deeper”. Or as John Ionides puts it “Even 1% improvement in the yield and translation of useful discoveries effected through better research practices reflects value equivalent of many Nobel or Breakthrough prizes.”
Why fund Registered Reports advocacy? Registered Reports are a new way of doing and publishing research, that might lead to a quite radical change in scientific practice. The new format is a partial solution to the replication crisis in science. How do Registered Reports work? Initially, only the theoretical motivation for study and a very precise methodology are submitted for peer review to the journal– before any data is collected or analysis is performed. The paper is then accepted, rejected or accepted if revisions to the methodology are made. Only then, the data is collected and the analyses are performed. The paper will be published regardless of the outcome e.g. whether a drug worked or not. Here we argue that this approach is superior to traditional publishing. A more widespread adoption of Registered Reports would change scientific practice for the better through increased replicability of studies across many scientific disciplines, more theory-driven science, and potentially simplified grant-making. Advocacy for Registered Reports is relatively neglected by other funders and there’s unique window of opportunity to advocate for wider adoption of the format by more scientific journals. Despite some push-back from vested interests in the publishing industry, we believe that advocacy is tractable, and there is a chance that with a little push through increased funding, Registered Reports might become the gold standard for hypothesis-driven science.
Why fund Chris Chambers? Professor Chris Chambers is a tenured Professor at Cardiff University, UK, and the co-inventor of the Registered Reports formats. He is already doing some Registered Reports advocacy but is time-constrained due to his university teaching load. Buying him out of his teaching and administrative work at the University would free up his time, so he could readily do more advocacy and introduce the Registered Reports to more and higher prestige journals. Our analysis suggests that funding this advocacy effort will lead to a cost-effectiveness of roughly $365-1460 per journal adopting the Registered Reports format, though this should not be taken literally. If a few hundred high-quality journals would sign up for the format, we believe it is not inconceivable that this could lead to non-linear increases in the adoption of the format and cause a paradigm shift in scientific culture, where the Registered Reports would be seen as the gold standard for hypothesis-driven science. We believe this is where most of the value of this project lies. We believe that this project has at the very least £112,000 ($150,000) room for more funding to free Chambers from teaching administrative duties for two years, but that the project could productively absorb even more so that funding is covered for several years. We close with a section in which we summarize some risks, reservations, and drawbacks of funding this grant.
How do Registered Reports work?
The Center for Open Science describes how the Registered Report format works in detail:
Authors of Registered Reports initially submit a Stage 1 manuscript that includes an Introduction, Methods, and the results of any pilot experiments that motivate the research proposal. Following assessment of the protocol by editors and reviewers, the manuscript can then be offered in principle acceptance, which means that the journal virtually guarantees publication if the authors conduct the experiment in accordance with their approved protocol.
With this in hand, the researchers then implement the experiment. Following data collection, they resubmit a Stage 2 manuscript that includes the Introduction and Methods from the original submission plus the Results and Discussion. The Results section includes the outcome of the pre-registered analyses together with any additional unregistered analyses in a separate section titled “Exploratory Analyses”.
Authors must also share their data on a public and freely accessible archive … and are encouraged to share data analysis scripts. The final article is published only after this process is complete. A published Registered Report will thus appear very similar to a standard research report but will give readers confidence that the hypotheses and main analyses are free of questionable research practices.
Our full, in-depth analysis is available here.
Watch a talk by Professor Chris Chambers on Registered Reports
Watch Chris Chambers give a talk on Registered Reports:
Listen to a Podcast with Professor Chris Chambers
Want to hear an in-depth discussion of Registered Reports? Listen to an episode of the “Everything Hertz” podcast with Chris Chambers about Registered Reports, co-hosted by Dr. Dan Quintana and Dr. James Heathers. Everything Hertz is “a podcast by scientists, for scientists. Methodology, scientific life, and bad language.”
Endorsements for the Registered Reports format
Professor Dorothy Bishop, Wellcome Trust Principal Research Fellow University of Oxford:
“Registered reports contrast with the conventional publishing model, where researchers may be left hoiking their paper around different journals to be greeted by an endless stream of reviewer criticisms. And this tends to happen when the funds have run out, the postdoc has moved on, and enthusiasm for the project is evaporating.
Perhaps the most important benefits are for science itself. With registered reports, there is no scope for publication bias
Professor Tahira Probst, Editor-in-Chief, Stress and Health:
“Scholars and academicians seek to produce cutting‐edge research that contributes to science and knowledge. Unfortunately, the intense pressure that accompanies this goal can actually erode the quality and rigor that should be at the heart of science. We launched our Registered Reports Initiative to maintain our focus on rigorous high-quality research while simultaneously circumventing some of the pitfalls of traditionally published research, such as publication bias against null findings and replication research. While the Registered Reports mechanism has been used in fields such as medicine, biology, and neuroscience, this publishing mechanism is only beginning to emerge within many of the disciplinary fields that contribute to Stress and Health, including psychology, organizational behavior, and occupational health. By leading the way, we hope to contribute toward the ultimate goal of inclusive and open science.
Estimates suggest over half of funded research projects fail to result in published findings. This is a serious waste of scarce research funding, as well as the time and talent devoted to those projects. While many reasons for this likely exist, the pressure to publish statistically significant findings seriously undermines our ability to collectively learn from prior research (including null findings). Registered Reports can help address this by promoting more inclusive research publishing, since submissions are evaluated solely on the quality and merit of the research idea and proposed methodology, rather than the statistical significance of the eventual findings.”
Professor Allen Moore, Editor-in-Chief, Ecology and Evolution:
“We live in an age of information, but the quality of information is variable. Science is not immune to the growing suspicion that information that is provided may be selective or that there may be unintended (or even intended) biases in what is available. Scientists are increasingly seen as working toward a personal agenda rather than working towards the public good.
In academia, this is a critical problem as we are dependent on the public, we work for the public good, and we are increasing public knowledge. We can improve confidence in our work by transparency – rather than hiding the process by which we reach conclusions, open the curtain.
Registered reports are an outstanding way to provide transparency. We can also educate on the process. Finally, the added work is actually minimal, as the best research is designed before it is undertaken, not during or after.”
Dr Emma Marsden and Professor Kara Morgan-Short, Associate Editors, Language Learning:
“We give three reasons why Language Learning introduced Registered Reports. 1) To incentivize replication research, as our systematic review found that only about 1 in 400 published articles in our field are replication studies. Registered Reports support would-be replicators via ‘In Principle Acceptance’. 2) Our review also illustrated the need to share materials and data to improve the amount and systematicity of replication research. More materials and data would become available through the registration aspect of Registered Reports. 3) As Associate Editors, we notice that many reviewers seek clarification about methods – and sometimes request changes that are impossible to address retrospectively. Registered Reports allow for methodological modification through peer review before methods are “locked in”.
Kaylin Ratner, Ph.D. Student at Cornell University:
“I would absolutely submit more Registered Reports to Royal Society Open Science. I was honestly surprised to learn that we were the first Registered Report to be accepted by the journal because the process was so smooth. Although I found the Registered Reports process to be more demanding upfront, I whole-heartedly believe that it was worth it. Once we had Stage 1 completed and approved, everything else felt like it fell into place and it was a welcomed change in contrast to the traditional publication process.”
Registered Reports should result in greater inclusiveness of researchers as they provide the opportunity to receive feedback on research design before data collection. This aspect of Registered Reports can support the development of more valid research designs by individuals in resource-poor positions, where access to feedback from colleagues and from conference attendance may be limited. Improved research designs would enhance these researchers’ ability to publish in high-visibility journals. Registered Reports may also be more inclusive theoretically and methodologically. This is because ‘minority’ or ‘controversial’ approaches are given more of a chance to reach publication as IPA prevents theorists from ‘rejecting’ a manuscript when data don’t align with their own standpoint.”
Apply for a PhD position with Professor Chris Chambers
If you’re looking for a PhD position, you can apply for a Ph.D. co-supervised by Chris Chambers on “Meta-scientific studies of reproducibility”. See more info below and apply here.
(There might be related jobs within meta-research at METRIC, the Meta-research Innovation Center at Stanford University, or with members of the Meta-research Center at Tilburg University in the Netherlands.). You can read more about the social impact of becoming an academic researcher at the 80,000 Hours ethical career guide.
This 3 year PhD studentship, part funded by the UK Reproducibility Network (UKRN), will focus on meta-scientific studies of reproducibility. The research will include a particular focus on assessing the efficacy of recent interventions that are intended to improve research quality and reliability, such as Registered Reports (https://cos.io/rr/), the Transparency and Openness (TOP) guidelines (https://cos.io/top/), Badges for Open Practices (https://cos.io/our-services/open-science-badges/) or related initiatives. Potential projects might include real-world randomised trials of specific interventions within targeted environments (e.g. academic journals), as well as observational studies and simulations to contrast different methods for improving reproducibility (e.g. multiverse analysis compared with study preregistration).
The student will be supervised by Chris Chambers and Candice Morey in the School of Psychology, Cardiff University, and by Marcus Munafò at the University of Bristol. The student will be based in Cardiff and will become one of a network of doctoral students supported by the UKRN, based across Cardiff, Bristol, Oxford and Edinburgh.
The studentship will commence in October 2019 and will cover your tuition fees (at UK/EU level) as well as a maintenance grant. In 2018-19 the maintenance grant for full-time students was £14,777 per annum. As well as tuition fees and a maintenance grant, all School of Psychology students receive conference and participant money (approx. £2250 for the duration of the studentship). They also receive a computer and office space, additional funding for their research, and access to courses offered by the University’s Doctoral Academy and become members of the University Doctoral Academy.
Full awards (fees plus maintenance stipend) are open to UK Nationals, and EU students who can satisfy UK residency requirements. To be eligible for the full award, EU Nationals must have been in the UK for at least 3 years prior to the start of the course for which they are seeking funding, including for the purposes of full-time education. As only one studentship is available and a very high standard of applications is typically received, the successful applicant is likely to have a very good first degree (a First or Upper Second class BSc Honours or equivalent) and/or be distinguished by having relevant research experience.