Likelihood of Finding Null Effects over Time
A few days ago, we choose an interesting paper to read in our bi-weekly journal club. The paper, “Likelihood of Null Effects of Large NHLBI Clinical Trials has Increased Over Time” 1, as its title suggests, more clinical trials are failing to find significant results. Researchers investigated the effect of pre-registration on the frequency of finding significant results in large NHLBI studies published from 1974 to 2012. Although very cautionary, they’ve concluded that most studies after 2000 ended up failing to reject the null hypothesis. The Discussion section — elegantly — listed several other possible factors responsible for this result. Fig 1. summarizes all studies subject to the meta-analysis. The plot clearly shows that after 2000, most studies failed to reject the null hypothesis. The visualization is on point and it clarifies the reasoning behind the aforementioned conclusion.
While reading and discussing the paper, we noticed a large gap between 1995 and 2000. The gap suggests that not many studies have been performed in this period. We also noticed that the plot does not show the starting year of the study. So, it is likely that several studies are being started between 1995–2000; however, it’s not possible to see this in the plot. Then, we asked, if there are studies that are started before 2000, have they pre-registered?
This is an important question as it might affect the conclusion of the paper. If studies that are failing to reject the null after 2000 have been started before the mandatory pre-registration, it is possible that the pre-registration is not the explaining factor for decrease in the number of null cases after 2000.
So, I’ve tried to re-create the plot. Below, I’ve plotted the same data with a dumbbell plot. I drew a line between the starting year of a study and its published year. Fig 2. suggests that most papers that failed to reject the null — after 2000 — are actually started before 1995 where the pre-registration was not necessary.
It’s very difficult claim that the published result is false and I am not at all trying to convey this message. However, I find it interesting that a small addition to the plot can provide a significant clarification.