Arguments - Quotes

Permutation statistical methods are a paradox of old and new. While permutation methods pre-date many traditional parametric statistical methods, only recently have permutation methods become part of the mainstream discussion regarding statistical testing.

Permutation tests are often described as the gold standard against which conventional parametric tests are tested and evaluated.

Early statisticians understood well the value of permutation statistical tests even during the period in which the computationally-intensive nature of the tests made them impractical.

With the advent of high-speed computing, permutation tests became more practical and researchers increasingly appreciated the benefits of the randomization model.

In a permutation statistical test the only assumption made is that experimental variability has caused the observed result. That assumption, or null hypothesis, is then tested.

Permutation tests are data dependent, in that all the information required for analysis is contained within the observed data set[...]. Permutation tests do not assume an underlying theoretical distribution [...]. Permutation tests do not depend on the assumptions associated with traditional parametric tests, such as normality and homogeneity [...].

Permutation tests are ideal for very small data sets, when conjectured, hypothetical distribution functions may provide very poor fits [...]

Appropriate permutation tests are resistant to extreme values, such as are common in demographic data [...]

[...] permutation tests were understood by many researchers to be superior to conventional tests as permutation tests were datadependent, did not depend on the assumptions associated with classical tests, were

appropriate for use with either an entire population or a nonrandom sample, and provided exact probability values.

The importance of the permutation approach in resolving a large number of inferential problems is well documented in the literature, where the relevant theoretical aspects, as well as the extreme effectiveness and flexibility, emerge from an applicatory point of view.

Under the Fisher–Pitman permutation model, no knowledge of the standard error is required, random sampling is not necessary, distributional assumptions are irrelevant, and permutation tests are completely data-dependent.

[...] the structure of the test statistic is no longer required if the assumption of normality is removed.

Mielke P. W., Berry K. J. Jr. (2007) Preface.

Permutation tests are often termed “data-dependent” tests because all the information available for analysis is contained in the observed data set. [...] Thus, permutation tests are distribution-free tests in that the tests do not assume distributional properties of the population.

Permutation methods are distribution-free, allow us for quite efficient solutions when the number of cases is less than the number of covariates and may be tailored for sensitivity to specific treatment alternatives providing one-sided as well as twosided tests of hypotheses.

Permutation tests permit us to choose the test statistic best suited to the task at hand. This freedom of choice opens up a thousand practical applications, including many which are beyond the reach of conventional parametric statistics. Flexible, robust in the face of missing data and violations of assumptions, the permutation test is among the most powerful of statistical procedures. Through sample size reduction, permutation tests can reduce the costs of experiments and surveys.

A large number of univariate problems may be usefully and effectively solved using traditional parametric or rank-based nonparametric methods as well, although under relatively mild conditions their permutation counterparts are generally asymptotically as good as the best parametric ones.

[...] permutation inferences are so important for both theoretical and application purposes, not only for their potential exactness.

The population model is rife with assumptions that are seldom satisfied in practice and are often inappropriate for the lower levels of measurement, e.g., independence, random sampling from a parent population, an underlying Gaussian distribution for the target variable in the population, and homogeneity of variance (and covariance,

when appropriate). In this book, the permutation model is used almost exclusively as it is free of any distributional assumptions, does not require random sampling, is completely data-dependent, provides exact probability values, and is ideally suited for the analysis of small samples.

The value of permutation statistical tests was recognized by early statisticians, even during periods in which the computationally intensive nature of permutation tests made them impractical.

[...] when one considers the whole problem of statistical inference, that is of tests of significance, estimation of treatment differences and estimation of the errors of estimated differences, there seems little point in the present state of knowledge in using [a] method of inference other than randomization analysis (Kempthorne, quoted in Ludbrook and Dudley) [719, p. 966].

Consequently, in the 21st century permutation statistical methods have become both feasible and practical and have found applications in diverse fields [...]

An attraction of the randomization test approach to hypothesis testing is its versatility. Randomization tests can be carried out for many single-case designs for which no stan dardrank test exists.

In real applications, random sampling, on which the parametric methods are based, is rarely achieved. Hence often the unconditional inferences associated with parametric tests are not applicable. In these situations permutation tests are suitable solutions.

For very small-n designs (less than 10 observations per treatment condition), the use of randomization tests is strongly recommended.

Fisher was a virtuoso performer on the primitive mechanical calculators of the time, but even he could make little use of randomization tests. In fact, they were little used until the widespread availability of fast computing made it feasible to make numerous reorderings and resamplings of data. In the 1970s, for example, Efron (1982) used the computing power then available to extend the randomization test approach to other resampling methods, such as the bootstrap. He commented that if we made as good use of our computing equipment as Fisher made of his, statistics would be in a very different place by now.