WWW quotes - Quotes

Permutation tests
Quotes
Title
Go to content
Quote
Link
They are employed in a large number of contexts: Oncologists use them to measure the efficacy of new treatment options for cancer. Google uses them to determine which color of blue (e.g. this blue vs this blue) is most effective for outgoing links. And entomologists use them to study the sex habits of flies.
With the increasing availability of inexpensive large-scale computational resources and openly shared, large datasets, permutation methods are becoming popular in neuroimaging due to their flexibility and ease of concern about yielding nominal error rates than parametric tests, which rely on assumptions and/or approximations that may be difficult to meet in real data.
[...] the permutation test perfectly represents our process of inference because our null hypothesis is that the two treatment groups do not differ on the outcome [...]
https://thomasleeper.com/Rcourse/Tutorials/permutationtests.html

Permutation tests, which I’ll be discussing in this post, aren’t that widely used by econometricians. However, they shouldn’t be overlooked.
Permutation tests date back to the 1930’s, and were first proposed by Fisher (1935), Pitman (1937a, 1937b, 1938) and others. This is certainly not a new idea! However, as you can imagine, in the 1930’s these tests could be used only with very small samples and this limited their appeal to some degree.
Permutation tests are "exact", rather than asymptotic (compare with, for example, likelihood ratio tests). So, for example, you can do a test of means even without being able to compute the distribution of the difference in means under the null; you don't even need to specify the distributions involved. You can design a test statistic that has good power under a set of assumptions without being as sensitive to them as a fully parametric assumption (you can use a statistic that is robust but has good A.R.E.).
Unfortunately, a lot of statistical tests require complex assumptions and convoluted formula. This is especially true of those methods taught in introductory courses, giving the false impression that experimental design is boring and unintuitive. But fret not, my valued reader - not all tests are so bad! In what follows, I present a visual explanation for the permutation test: an awesome nonparametric test that is light on assumptions, widely applicable, and very intuitive.
Permutation tests
Back to content