Idea - Quotes

Quote	Source
Idea
Five Steps to a Test 1. Analyze the problem—identify the hypothesis, the alternative hypotheses of interest, and the potential risks associated with a decision. 2. Choose a test statistic. 3. Compute the test statistic. 4. Determine the frequency distribution of the test statistic under the hypothesis. 5. Make a decision using this distribution as a guide.	Good (2005) p. 8.
• Is it exact? That is, can we make an exact determination of the probability that we might make an error in rejecting a true hypothesis? • How powerful a test is it? That is, how likely is it to pick up actual differences between treated and untreated populations? Is this test as powerful or more powerful than the test we are using currently? • Is the test admissible? That is, is there no other test that is superior to it under all circumstances? • How robust is the new test? That is, how sensitive is it to violations in the underlying assumptions and the conditions of the experiment?	Good (2005) p. 14.
1. Permutation statistical methods are entirely data dependent, in that all of the information required for analysis is contained within the observed data set. 2. Permutation statistical methods do not depend on the assumptions associated with traditional parametric tests, such as normality and homogeneity of variance. 3. Permutation statistical methods provide exact probability values based on the discrete permutation distribution of equally-likely test statistic values, rather than an approximate probability value based on a theoretical distribution, such as a normal, chi-squared, t, or F distribution. 4. Although permutation statistical methods are suitable when a random sample is obtained from a specified population, permutation methods are also appropriate for nonrandom samples, such as are common in everyday research. 5. Permutation statistical methods are appropriate for analyzing entire populations, as permutation methods are not predicated on repeated random sampling from a specified population. 6. Permutation statistical methods can be defined for nearly any selected test statistic. Thus, researchers have the option of using a wide variety of statistics, including the majority of conventional statistics utilized in classical statistical approaches. 7. Permutation statistical methods are ideal for small data sets, when hypothetical distribution functions may provide very poor fits. 8. Appropriate permutation statistical methods are highly resistant to extreme values, such as are common in demographic data, e.g., age at first marriage, income, and so on. Consequently, the need for any data transformation is mitigated in the permutation context and in general is not recommended, e.g., square root, logarithmic, arc cosine, and other transformations, including the conversion of raw scores to ranks. 9. Permutation statistical methods provide data-dependent statistical inferences only to the actual experiment or survey that has been analyzed, and are not dependent on knowledge of a super population. On the other hand, if random sampling from a specified population has been accomplished, then permutation tests can provide inferences to the population.	Berry et al. (2018) p. 628.