What does a p value actually mean?

The definition is awkward because it is conditional on something you do not believe. It assumes the null hypothesis is exactly true, assumes your model is correct, and then asks how surprising your data would be. A small p value tells you the data sit uncomfortably with that whole package, and it does not tell you which part of the package is wrong.

That is why the common misreadings are wrong in a specific way. A p of 0.03 does not mean a 3 percent chance the null is true, and it does not mean a 97 percent chance your hypothesis is right. It says that data this extreme would occur 3 percent of the time in a world where the null held and your assumptions were satisfied.

A small p value also does not tell you which assumption failed. Greenland and colleagues list the candidates directly: the test hypothesis may be false, some other model assumption may be wrong, the result may be random error, or the analysis you reported may have been chosen after many others were tried (Greenland et al., 2016, p. 341). Crossing 0.05 on its own does not establish that the tested hypothesis is false (Greenland et al., 2016, p. 346).

How much a given p value should move you depends on what you tested. Ioannidis works through a genome-wide association study screening 100,000 polymorphisms for schizophrenia, of which roughly ten are truly associated. With 60 percent power and a 0.05 threshold, a result that just crosses 0.05 multiplies the probability that any one polymorphism is genuinely associated by about twelve, which still leaves it at 12 in 10,000 (Ioannidis, 2005, p. 699). The same number carries very different weight in a study with one prespecified comparison.

The 0.05 threshold is a convention, not a discovery. Treating 0.049 and 0.051 as categorically different results is a habit that professional statistical bodies have argued against for years, and reporting the exact value with the effect estimate avoids the problem entirely.

The practical consequence for your thesis is that a p value should never be the headline. The headline is the size of the effect and how precisely you estimated it, and the p value is a supporting detail.

What does a non-significant result tell me?

That your data did not provide strong evidence against the null, which is not the same as evidence that the null is true. A non-significant result in a small study is usually uninformative. Report the estimate and its confidence interval, which shows what effects remain plausible.

The interval does the work here. A non-significant result whose interval runs from a large negative to a large positive effect tells you almost nothing. A non-significant result whose interval is narrow and close to zero is genuine evidence that any effect is small, and that is a finding worth reporting.

Precision, not effect size, is often what separates two results that look like they disagree. Greenland and colleagues describe two randomized trials with the same observed treatment difference of 3. Trial A has a standard error of 2 and gives p = 0.13. Trial B has a standard error of 1 and gives p = 0.003 (Greenland et al., 2016, p. 343). The two trials agree on the estimate and differ only in how precisely they measured it, so reading one as a null result and the other as a positive finding misreports both.

Absence of evidence phrasing matters in the write-up. "No effect was found" invites a reader to hear "there is no effect", and "no evidence of an effect was found, with the interval consistent with effects up to X" says what you actually established.

Should I report effect sizes instead?

Report both, and lead with the effect size and its interval. The p value answers a narrow question about compatibility with one hypothesis. The effect size answers the question a reader actually has, which is how large the difference or relationship is.

Report the effect in the units of your measure where you can, since a difference of four points on a familiar scale is more interpretable than a standardized coefficient. Add the standardized version too when comparing across studies is relevant.

Report the exact p value alongside them. Greenland and colleagues recommend writing p = 0.015, p = 0.06, or p = 0.70 rather than collapsing the number into a significant or not significant label (Greenland et al., 2016, p. 342). Kass and colleagues point at the habit behind that collapse, the belief that a finding is unworthy of publication unless p falls below 0.05, and call it a major source of criticism of statistical practice (Kass et al., 2016, p. 6).

Most journals in most fields now require effect sizes and intervals, and many reporting guidelines specify them. Writing the results this way from the start saves a revision round later.