Foundations of statistics

The Foundations of statistics concerns the epistemological debate in statistics over how one should conduct inductive inference from data. Among the issues considered in statistical inference are the question of Bayesian inference versus frequentist inference, the distinction between Fisher's "significance testing" and Neyman–Pearson "hypothesis testing", and whether the likelihood principle should be followed. Some of these issues have been debated for up to 200 years without resolution.

Bandyopadhyay & Forster describe four statistical paradigms: "(1) classical statistics or error statistics, (ii) Bayesian statistics, (iii) likelihood-based statistics, and (iv) the Akaikean-Information Criterion-based statistics".

Savage's text Foundations of Statistics has been cited over 12000 times on Google Scholar. It tells the following.

It is unanimously agreed that statistics depends somehow on probability. But, as to what probability is and how it is connected with statistics, there has seldom been such complete disagreement and breakdown of communication since the Tower of Babel. Doubtless, much of the disagreement is merely terminological and would disappear under sufficiently sharp analysis.

In the development of classical statistics in the second quarter of the 20th century two competing models of inductive statistical testing were developed. Their relative merits were hotly debated (for over 25 years) until Fisher's death. While a hybrid of the two methods is widely taught and used, the philosophical questions raised in the debate have not been resolved.

Fisher popularized significance testing, primarily in two popular and highly influential books. Fisher's writing style in these books was strong on examples and relatively weak on explanations. The books lacked proofs or derivations of significance test statistics (which placed statistical practice in advance of statistical theory). Fisher's more explanatory and philosophical writing was written much later. There appear to be some differences between his earlier practices and his later opinions.

...
Wikipedia