The result at the end of our statistical analysis is a $p$ value. We get a $p$ value for each hypothesis test / inferential statistic we calculate. This is the primary value that we use to decide whether to reject our null hypothesis. It is also one of the most important statistics to report.

The $p$ value describes the probability that the results we observed were the result of chance. As it is a probability, so it must always be in the range $0 \lt p \lt 1$.

The $p$ value gives us the likelihood of making a Type 1 Error. This type of error is a false positive - where we conclude there is an effect when there is no effect.

The $p$ value is calculated from test statistic and degrees of freedom.

The $p$ value is compared against a significance level to decide whether the chance that we have observed an effect is large enough to conclude that we really have observed an effect. We must set the significance level for each statistical test we conduct. Typically this value is no larger than $p = 0.05$, meaning it is normal to conclude that we have observed an effect when there is less than a 5% chance of a false positive.

If the $.05$ level is achieved ($p$ is equal to or less than $.05$), then a researcher rejects the $H_0$ and accepts $H_1$. If the $.05$ significance level is not achieved, then the $H_0$ is retained

We can think of calculating a hypothesis test against an alpha (significance level) of 0.05 as like rolling a 20 sided dice. There is a 1 in 20 chance (0.05) of rolling a 20. Here rolling a 20 represents the situation where we have a false positive - where we incorrectly conclude that we observed an effect.

From probability, we can calculate that the probability of getting at least one 20 if we roll the dice twice as being $0.05+0.05 - (0.05 \times 0.05) = 0.0975$. This is almost double the chance of getting a 20 if we roll the dice only once! The probability of getting at least one 20 increases for every time we roll the dice. Similarly, with hypothesis testing, the probability of getting at least one false positive increases with every hypothesis test run.

In any situation where we are testing (effectively) the same hypothesis multiple times - and where we would report having detected an effect based on any individual test finding an effect - we need to reduce our alpha to adjust for this increased probability of a type 1 error (false positive). (This is also why it is important for null results to be published - if only positive results are published, there will be an inflated false-positive rate.)

The degrees of freedom for a statistic is the number of values used in its calculation that would be able to vary without changing the value of the statistic itself. This can be confusing to understand, so let's consider a simple example with the calculation of the arithmetic mean of a sample of four numbers:

Here we have four values, or data points, involved in the calculation: 3,4,6, and 7. Were we to report the mean of this set of numbers (which as we can see is 5), we could say that the mean is 5, and there are 3 degrees of freedom. To understand why there are three degrees of freedom, you need to imagine for a moment that you do not know what data this statistic was calculated from.

Now assuming that you know that the mean is 5, and that the sample size was 4 (i.e., we found the mean of four numbers) only three of the four data points are free to vary independently. This is because while three of the values could have been any number, the fourth value is completely determined by the previous three and the mean. In other words, If I gave you three values and the mean, you could work out the fourth value. Algebraically, you can see this as the following problem, where you can solve for $x$:

Thus the value of $x$ in this case is not free to vary, but is completely specified by the other values. There are three remaining data points that are free to vary, thus our mean has three degrees of freedom, written 3 d.f.

\( \text{d.f.} \) affects the interpretation of some statistics. There are rules for how to calculate d.f. for each statistical test

The sampling distribution is the statistical distribution that your observed value comes from. For example if we sample individuals, our sampling distribution is the same as our population distribution. In the example above, we sampled individual fish, so our sampling distribution was the same as our population distribution, which was a normal distribution.

However, often the observed value is not an individual measurement, but the mean of a group of measurements. Means of values drawn from a given distribution have a different distribution to the values themselves. Consider that it is much more likely to get a single extreme value than to average several values and get an extreme value for the mean. Therefore, when we sample group means of size $n$, our sampling distribution is a distribution of means of groups of size $n$.

According to the Central Limit Theorem, for large sample sizes, the sampling distribution for means approximates a normal distribution with a mean equal to the population mean, and a standard deviation equal to \( \frac{\sigma}{\sqrt{n}} \). This applies even when the population distribution is not itself normal.

Standard error is the standard deviation of our sampling distribution. For our fish example, our standard error would be the standard deviation of our data (assuming our individuals are normally distributed).

When comparing group means, we need to adjust our standard error as our sampling distribution is no longer normal. It is a distribution formed of means of groups of size $n$. The standard deviation of this distribution is our original $\sigma$ divided by the square root of the group size. We can still calculate a $z$ statistic (as it's still in standard deviations).

When comparing differences in group means, again we have a different sampling distribution, so we need a different formula for our standard error:

Let us begin with our basic formula for an inferential statistic:

Consider the situation where we observe a single data point and we are comparing it to the population mean. Our observation is of a data point, $x$. Our expectation would be that the value is the population mean, $\mu$. The typical variation is given by the population standard deviation, $\sigma$.

Imagine we randomly sampled a normal distribution and calculated z statistics for single values, for samples, or for comparing two sample means using the $z$ statistic formulas above. We could then plot the probability distribution that the z statistic follows. This is called the $z$ distribution.

If we generate a random $z$ statistic - i.e. we perform a random hypothesis test - we might expect that the majority of the time we are going to see a small value for $z$, because we sampled from our population distribution. This is because most of the time differences that we observe between our sample mean and the population mean are going to be small. Extreme values for $z$ are going to be uncommon, because we only get a large value for z when we happened to have selected a sample with an unusually large or small mean.

As it happens, the $z$ distribution is a normal distribution. The mean is 0 (because the average difference that we would observe between sample and mean is 0). The standard deviation is 1.

Now we can compare the value we have computed for $z$, and calculate the probability that we would observe that value by chance. If the probability is low, that suggests it is unlikely that our observation (which we used to calculate $z$) is drawn from the population distribution, the value is likely drawn from some other distribution. In other words, where the probability of getting our $z$ value is low, it is likely because our sample is significantly different to the population.

We rarely know the population standard deviation, and thus we cannot often use a $z$ test. More often, we know the standard deviation of a sample. This standard deviation is only an approximation - though the larger the sample size used to calculate the standard deviation, the better that approximation is.

Because of the error inherent in using an approximation of standard deviation, the values that we derive are different. This is most evident when sample sizes are smaller than 30. When we use the approximation of standard deviation, the resulting statistic is called a $t$ statistic.

Like Z tests, T tests assume data follows a normal distribution. This makes it a parametric test. It similarly assumes representative, randomly selected samples and similar variance between groups.

As the $t$ statistic is calculated with an approximation of standard deviation that depends on sample size (which is closely related to degrees of freedom), the $t$ distribution is dependent on the number of degrees of freedom in the calculation.

When the distribution of the $t$ statistic for a given sample size is plotted, we see that, where degrees of freedom are low, the distribution has thicker tails. When the degrees of freedom are high, the distribution approximates the $z$ distribution.

(Note: In the interactive example embedded on the slides, $v$ is degrees of freedom)

For each test statistic, we can calculate the value that corresponds to a particular probability. For example, we can calculate the z statistic where the probability of generating a value at least as extreme as it is 0.05. If our significance level is set to 0.05, we are able to compare our calculated $z$ against this value to check if our result is significant at the 0.05 level.

Find t tables at t-tables.net

Effect size is usually calculated using the formula for Cohen's $d$.

This is the observed difference between the samples as a proportion of the standard deviation. If the standard deviation varies between groups, the pooled standard deviation must be used:

Where $s_1$ and $s_2$ are the standard deviations of $x$ and $y$ respectively.

Effect size is a unitless measure that can be used to compare the effect sizes of different studies. A very approximate rule of thumb - where there is no more specific data available is:

To determine if sample means come from same population, use 5 steps with inferential statistic

1. State Hypothesis

$H_0$: there is no difference between 2 means; any difference found is due to sampling error

$H_1$: there is a difference between the two means; the samples are drawn from different distributions.

Results are stated in terms of probability that $H_0$ is false. Usually the goal of a study is to reject $H_0$.

2. Level of Significance

Decide on the alpha level, or level of significance. This is the probability of observing a result by chance, or observing a result if two samples were drawn from the same population. This is usually set to $\alpha = 0.05$. The smaller the value of $\alpha$ the more confidence we require to conclude that we have found a true effect.

3. Computing Test Statistic

Use a statistical test formula to derive an appropriate test statistic (e.g. $z$ value, $t$ value, or $F$ value). We need to select a test statistic based on appropriate assumptions about our data.

4. Obtain Critical Value

A value derived from the test statistic distribution for a given number of degrees of freedom, and the significance level $\alpha$. We compare our calculated test statistic to this critical value to determine if findings are significant. If our test statistic is larger than the critical value, we reject $H_0$.

5. Reject or Fail to Reject $H_0$

Calculated value is compared to the critical value to determine if the difference is significant enough to reject Ho at the predetermined level of significance

If test statistic is greater than the critical value then we reject $H_0$. If test statistic is less than the critical value then we fail to reject $H_0$. Rejecting $H_0$, supports $H_1$, but it does not prove $H_1$.

Most statistics assume your data are normally distributed. If they are use a parametric test if not use a non-parametric test.

How to decide? Test for normality - use the Shapiro-Wilk or the Kolmogorov-Smirnov test. S-W generally considered to be better.

For both if $p \gt 0.05$ we assume a normal distribution