Notes about some risks you run

The aim of this site is to share some information and sources for your further investigation. It may be that after reading it you feel inclined to abandon the practice of NHST. That comes with risks. Here are some.

A personal risk you may run into is a slowdown in your career. Why? Because use of significance testing can make it easier for papers to be accepted in some places, since it's a widely used magic password that conveys that “what you found is really real and a thing that really exists and is true”.

Quoting Salsburg 1985: “Invocation of the religious dogmas of Statistics will result in publication in prestigious journals. This form of Salvation yields fruit in this world (increases in salary, prestige, invitations to speak at meetings) and beyond this life (continual references in the citation indexes)”.

It does not logically follow that refusal to invoke the dogmas would prevent such achievements. But it suggests uncertainty. Straying from the path exposes you to volatility. You may be cheered, or you may be rejected.

So if you decide to opt out of this ritual, you may have pieces of research rejected. Or not taken seriously. Or having colleagues shaking their heads wondering: “What made this bright young person deviate from the path, what with their reluctance in evaluating the significance of their research? How will journals know whether their finding is real? Why are they being difficult?”

This issue seems to be less prevalent than decades ago. We know that because someone checked the data and the difference is significant (p = 0.048). That said, if you choose to abandon NHST, your work may not be accepted here or there, nor taken seriously by some people, depending on where you are, your seniority, your field, your advisors, the journals you submit to, and luck.

We should create environments where less people be faced with this problem, where incentives are aligned. Meanwhile, you may face some personal dilemmas, frictions and setbacks.

And here is a letter for you.