TechnoRamble: June 2016

next blog post on Linear Regression

Lesson 1

Data as evidence
Anecdotal evidence - is a one-off data point
Statistical studies
What is the population of interest and what is the sample ?
Studies of a sample of the data to make conclusions about the population
Data basics
Observations (rows), variables (columns) and data matrix
Types of random variables -

Numerical and Categorical
Numerical -> Continuous (height) or Discrete (number of houses owned)
Numerical -> Interval (e.g. Dates, Temperature in Celcius) and Ratio (e.g. Mass, Temperature in Kelvin)
Categorical -> regular categorical(red, blue) or ordinal (very bad, bad, fair, good, very good)

Causation v/s Correlation.
Is one Random variable correlated or associated to another ? If so, then they are dependent variables, else independent. Correlation may be positive or negative.
Observational studies (merely observe phenomena and record data) and Experimental studies
Radomized, controlled experiments enable conclusions about causation, not observational.
Even if a correlation is found between x (explanatory variable) and y (response variable) in an observational study, three possibilities are there as regards causation :

x causes y
y causes x
a third unknown variable is causing both x and y

Sampling
Census ( whole data is the set studied) and Sample. Census apart from being expensive may not be a good idea because swathes of population may not respond due to some fears etc. and also population is constantly changing.
Checking if the soup is tasty enough by tasting only a spoonful is like studying a sample. If the pot contents are stirred properly (random sample) the sampled spoon is more representative of the whole pot even if it is a small spoon.
Types of Bias

Convenience bias - easily accessible people are more likely to be included in a sample
Non-response bias - sample is random but a whole category of people don't respond
Voluntary response bias - sample is not random. It is comprised of people who want to respond.

Bias prevents results from observational/experimental studies on samples being generalised to the population

Sampling Methods
Simple Random Sampling (SRS) - data points are selected such that each is equally likely to be selected.
Stratified sampling - To ensure that all categories are represented in the sample we may divide the population into categories and then pick randomly from each category (e.g. Male and Female)
Cluster sampling - First identify from the Population, the non-homogeneous, properly random (unlike stratified sampling) clusters, then randomly select a few clusters and then randomly sample from these clusters just like in Stratified sampling. This is done for economy reasons so that we don't need to incur the expense of randomly selecting from the whole population.
Experimental design
4 principles of experimental design : 1) control 2) randomise 3) replicate 4) block
control - compare treatment of interest to a control group
randomize - randomly assign subjects to treatment
replicate - collect a sufficiently large sample, or replicate the entire study
block - block for variables known or suspected to affect results. Block variables are the variables that can't be controlled (they are inherent properties of the subjects) but are still suspected to affect the outcome. e.g. gender may be suspected to affect the response variable, hence use stratified sampling based on Male and Female categories to form the treatment and controlled groups, so that both genders are properly represented in both groups and any result difference between the groups can then confidently be assumed not to be because of gender.
Random Sampling and Random Assignment

Orange people are generally slow readers and black are fast readers.

We want to find whether Serif fonts affect reading speed. We eliminate effects of the confounding variable i.e. other reasons for reading speed by using random sampling and random assignment.

Note here Random assignment means assignment to the treatment and control groups. Generalizable means results are generalisable to the population.

Example of Non random sampling is Volunteer studies, where the sample comprises of volunteers and are not randomly selected from the population.

Visualizing Numerical data
Scatter plots

Histogram

Histogram gives an idea of data density and especially useful to give the shape of the distribution (skewed, uni/multi-modal etc.)

Skew

Modality

Effect of bin width on histogram

Dot plot

Box plot

Useful to see outliers, medians and Inter Quartile ranges (percentile ranges)

Correspondence between box plot and histograms

Intensity plot

Measures of central tendency

Median and mean in skewed distributions

The tail portion pulls the mean with it.

Measures of Spread
Range (max -min) - Not too good
Variance
Standard Deviation
Inter-quartile range

Standard Deviation is the square root of the Variance
Variability and Diversity

Although set 1 is more diverse wrt color and mileage, set 2 has greater mileage variance

IQR is a better measure of spread than range, because it is the length of the middle quartile ( Q75 - Q25). Here Q25 is the variable value below which there are 25% data points and Q75 is the variable value below which there are 75% data points. Hence, the IQR does not include outliers which the range does.
Exploring Categorical Variables - describing distribution of a single categorical variable, relationship between two categorical variables and relationship between a categorical and a numerical variable
Bar plots ( similar to histograms) - rather than counts, percentage count may be used

Avoid Pie charts
Contingency table - shows counts/relative frequencies as percentages for two variables , one for the row and the other for the column of the table.

Segmented bar plot

Segmented bar plot with relative frequencies

Mosaic Plots

Side by side box plot

Inference

Typically, H0 says that the response variable is actually random wrt the explanatory variable, i.e. has no correlation with it. In a randomised controlled experimental study, the response variable values for the control group are used to get the mean and std dev and this distribution is H0. Then if the mean of the response variable values for the treatment group lies in the 2σ- interval (i.e 95% frequency interval. We get to choose this) we can say that there is no strong evidence to reject H0. We are not saying H0 is right. This method of inference is based upon looking at p( (data >= z-score) | H0) (also called p-value) where data is the mean of the response variable for the treatment group and seeing if this value is less than 5% which is the tail area.

Frequentist and Bayesian approach
Frequentist - Probability of an outcome is the proportion of times the outcome would occur if we repeated the experiment an infinite number of times
Bayesian - Probability of an outcome is a subjective degree of belief that the outcome will occur. This allows prior info to be integrated into the framework
Law of Large numbers

Gambler's fallacy - Although Law of Large numbers is true, it doesn't mean that the probability of the next experiment is biased toward ensuring it (assuming the trials are independent). i.e. if it is known that a coin is fair, and in three tosses we got a Heads, then the probability of getting a Heads on the fourth toss is still 0.5, and not any less just to ensure that Law of Large numbers is satisfied - any way 4 tosses are not too many.

Lesson 2

Kolmogorov's Axioms of Probability

Probability of any event is non-negative - p(E) >= 0
Addition Rule - If two events are mutually exclusive then the probability that either occurs is the sum of their probabilities.
The probability that some event from the Sample space ( the set of all possible outcomes for a trial) occurs is 1.0

Independence

Two events A and B are said to be independent if the occurrence of one doesn't change the probability of the other i.e.

p(A/B) = p(A)

p(B/A) = p(B)

p(A and B) = p(A).p(B|A) = p(B).p(A|B) = p(A).p(B) This is also the test (or definition) for independence.

p(A|B) is the probability of A given B i.e. probability of A conditioned on B

Independence is a symmetric relationship (but not reflexive and not transitive)

If two events are dependent then they are correlated

If two events are disjoint they are certainly dependent.

Determining dependence based on sample data

observed difference between conditional probabilities -> dependence -> hypothesis |----> if the difference is large, there is stronger evidence that difference is real |----> if the sample size is large, even a small difference can provide strong evidence of a real difference

Probability Trees and Bayes' Theorem

The encircled values can be used to find conditioned probabilities.e.g p(free/spam) = 35/60

as well as joint probabilities e.g. p(free and spam) = 35/100

The Marginal or Total probability of 'free' is

p(free) =

p(free and spam) + p(free and no-spam) =

p(free|spam).p(spam) + p(free|no-spam).p(no-spam)

Bayes' Theorem:

p(A and B) = p(B).p(A|B) = p(A).p(B|A)

p(A|B) = p(A and B) / p(B) = p(A and B) / (p( A and B) + p( ~A and B) )

p(A) is called the prior, and p(A|B) is called the posterior probability of A

Bayesian Inference

We evaluate claims iteratively as we get more and more data. We replace the prior by the posterior for the next iteration. The Hypothesis is 'probability of A having occurred'. And the process evaluates p(Hypothesis | data) iteratively.

Example

Let A = 'patient has HIV'. Let T = 'Elisa test is positive'. The data point is that patient A actually tested positive. Then,

p(A|T) = p(T|A).p(A)/p(T) = p(T|A).p(A)/(p(T|A).p(A) + p(T|~A).p(~A))

Here we first assume a prior for p(A). p(T|A) and p(T|~A) are known from elsewhere. Then knowing the data point, we calculate the new value (posterior) that A has occurred as above.

If the second data point is that the patient got retested and again tested positive, then we simply replace the prior by the posterior, and do the calculation for the new posterior again. This is how our belief in event A gets conditioned by more and more data.

A good prior helps, a bad prior hurts at first but matters less the more data you have.

More advanced Bayesian approaches offer flexibility not present in the frequentist approach.

Note how Bayesian inference is useful in Observational studies. Our earlier frequentist approach to Hypothesis testing was suitable in an Randomised, controlled, Experimental study setting.

Normal Distribution
The probability density function follows a bell curve - it is symmetric and unimodal. The mode, median and mean are the same and at the centre of the distribution graph.
The mean μ is the arithmetic average and std deviation σ is the root of the mean square error from the mean.
z-values or standard scores are normalised values of a random variable. For a random variable,x, z-value is (x-μ)/σ
z-values can be used to get the percentage of data points less than the value. From this we know that the 1σ-interval (z value -1 to 1)has probability 68.26% data points etc. We can get the relation of z and cumulative percentage values from charts

Normal probability plot
- used to determine if a set of data values represent a normal distribution. Data (z-values) are plotted on y-axis and the percentiles on the x-axis ( assuming normal distribution). If the scatter plot is a straight line then the distro is normal and it means there is a one-one correspondence between the data and the percentile.

This plot can also tell us about skewness and whether there is a long tail

Another way to assess whether a sample data is normal is to find its σ confidence intervals and see if the percentage frequency matches with the values found from the z-value charts. This is the 68-95-99.7 rule
Binomial Distribution -
Bernoulli trials - An experiment with a yes/no result .p is the probability of a yes, and (1-p) the probability of a no. The outcome of the trial is called the Bernoulli random variable and it is a Boolean.
The Binomial distribution is the probability function, mapping the number of successes out of the n trials (integers 0 to n) to its probability value.
For k successes, p(k) = p^k * (1-p)^(1-k) * kCn
Mean of a Binomial Distr = n*p
Std deviation = sqrt(n*p*(1-p))
As the number of trials in a normal distribution increases, it approaches the normal distribution
Normal approximation to a Binomial Distribution

n*p >= 10 and n*(1-p) >= 10
This is called the success-failure rule : A binomial distribution with at least 10 expected successes and 10 expected failures closely follows the normal distribution

Lesson 3 Statistical Inference

Sampling Variability and Central Limit Theorem (CLT)

Population mean μ , standard deviation σ . Sample mean x', and standard deviation s . As the Sample size increases, x' more and more approaches μ and s gets lesser and lesser than σ .

Note that the sample means, x', themselves form a distribution called the sampling distribution, with their std deviation being called standard error. As the size of each sample increases, SD(x') gets lesser and lesser than σ and mean(x') approaches μ

Central Limit Theorem

This is big. It means even if the distribution for population is not normal, the distribution for the sample means is, provided the sample size is large enough for each sample.

Confidence Interval

When we use a sample statistic ( mean) to guess the mean of the population, we can't be too sure. The confidence interval quantifies our confidence that the population mean is the sample mean. We say that the population mean is the 'sample mean with a confidence interval of XX%'. We can specify a 68 or 95 or 99.7% confidence interval.

Since we know that the sample mean is distributed around the population mean with a standard error of σ /sqrt(n) where n is the sample size, we have the equation

x' - z* . s/sqrt(n) < μ < x' + z* . s/sqrt(n)

where s, the sample std deviation has been for the population standard deviation, because of course we don't know the latter. z* is called the critical value of the CI.

For 68% confidence interval, z* is approximately 1, for 95%, 2, and for 99.7%, 3. The exact value f z* for 95% CI is determined below from the z-value chart

The margin of error is defined as z* . s/sqrt(n).

There is a slightly different way to think about this. If we have several samples of size n, and we calculate the means, x', of each then we know x' itself is a random variable which is normally distributed around μ with a standard deviation ( called standard error) of σ /sqrt(n). We can then say that 95% of these sample means are within +- 2*(standard error) of the true mean μ . This is the same as saying that 95% of the sample means have μ within +- 2*(standard error) of the sample mean i.e. that about 95% of those samples are such that their intervals contain μ . (The standard error is actually defined in terms of the population standard deviation σ which we don't know. ). Typically, CIs of 90%, 95%, 98% and 99% are used.
If we want be certain we capture the population parameter (μ ) we want to use a higher percentage CI - like 99%. This is the same as saying if we want to be accurate, we need a wide CI. If we use a sample size n that is large we'll get a very tight interval for this high confidence ( since standard error has sqrt(n) in the denominator). And we want a tight interval, otherwise our sample mean may not be very precise.

Example:

The first statement, a, says that the probability of Americans ( the population) spending 3.53 to 3.83 hrs relaxing is 95%. This is wrong, since it talks about the population distribution and we can only say that the probability that the population mean is in the interval 3.53 to 3.83 hrs is 95%.
Statement c is wrong because the population mean is fixed and does not move between two values.
Statement d is wrong because the sample average is 100% between 3.53 to 3.83 hours. It is exactly the middle of this interval.

Hypothesis testing for a single mean

The null hypothesis is that the true (population mean) is μ0. You test a sample and find the sample mean to be x'. You want to test if the obtained x' means the true mean is μ0 (null hypothesis is true) or whether you can reject the null hypothesis. The null hypothesis will be true if x' is within a say 95% interval centred on the true mean, which is the same as saying if 95% of the samples have μ0 within 2 standard errors on either side of their sample mean, x'. And we have seen this earlier

Note that HA: μ != null value is a two-sided hypothesis, i.e. we have to calculate the tail areas on both the sides of the mean. For the other two variants (< and >) only one side of the distribution.

Application of Hypothesis testing to other Point estimators

So far we have applied HT to only estimating population mean using sample mean for a data whose sampling distribution is near normal. But the HT method can also be applied to other point estimators

An important assumption for us to apply HT here is that the point estimates are unbiased. That the sample mean is an unbiased estimate of the population mean follows from the Central Limit Theorem.

Confidence Interval for nearly normal point estimates

point estimate +- z* . SE

Here SE is the Standard Error whose calculation will differ in general depending on the type of point estimate. For this lesson we assume it is given. For the sample mean point estimate we know what the calculation was ( σ / (sqrt(n)) ).

Examples

example 1

So we can say that we are 95% sure that between 30.3 and 35.7 % of all college graduates watch the Daily Show.

example 2

We need a population for the difference between the values for men and women.

Decision Errors

Type 1 and Type 2 errors .
Type 1 error is when you shouldn't have rejected the null Hypothesis (H0) because it was true, and you did. The probability for this is α, the significance value or the area under the tail beyond the CI limit.

Type 2 error is when you should have rejected H0 because it was false, but you didn't. The probability for this is β and it is not trivial to calculate this.

Power of a test is the probability of rejecting the null hypothesis when it should be ( i.e. when it is false)

Type 1 error :

Suppose we use a 5% significance level, then there is a 5% chance that we wrongly reject H0 when it is true, so the Type 1 error is 5%. In terms of probability

p(Type 1 error | H0 is true) = 0.05

here 0.05 is α . The larger the value of α the larger will be the Type 1 error.
If Type 1 error is dangerous or costly, then choose a small value for α.

If Type 2 error is dangerous or costly then increase the value of α since that will reduce the probability of not rejecting H0 when H0 is actually false.

While the significance level α used for CI and HT is directly related to the probability, p(Type 1 error | H0 is true) , indeed is equal to it, the probability

p(Type 2 error | H0 is false) = β

is not easy to calculate. But obviously,

p(H0 is rejected and hence no Type 2 error | H0 is false) = 1-β

If the true population average is very close to the null value then it will be difficult to detect a difference and reject H0. On the other hand, if the difference is large it is easier to detect the difference. So, β depends on this difference. The difference is called effect size(δ ).

Agreement between Hypothesis tests and CI

Hypothesis tests and CIs are very similar except that the former talks about the observed data point lying enough in the tail (whose probability is the area below the tail of the density curve for the null hypothesis) as to justify rejecting the hypothesis. The CI talks about the middle portion of the normal distribution curve around the sample mean (or other point estimate) as the centre and talks about the population statistic (which is being estimated) lying in that interval with a given confidence (95%, 90% etc.). So, in a sense CI and HT talk about complementary intervals of the density curve. But there is a caveat.

The third point requires some explanation. Here we use the sample data to find the sample mean ( or other point estimate) and we need the standard error of the sampling distribution to find the confidence interval for the population mean ( or other statistic) to lie in. That the population mean lies in the CI is the null hypothesis. Rejecting H0 means the population statistic lies outside the CI.

Statistical v/s Practical significance

The p-value for a large sample size will be much much smaller, so z-scores are pushed far away from the centre and for most values of x' that we find, they will lie far into the 5% significance area, lending a high statistical significance to the result that H0 is false. But if H0 is actually true, then most of the time results from the test will not say so.

This problem happened because μ and x' are very close to each other ( 50 and 49.5) i.e. the effect size is too small. So inflating the sample size misled the researcher in this case. The result was not practically significant.

The fifth edition of the APA (2001) Publication Manual states:

that it is almost always necessary to include some index of effect size or strength of relationship in your Results section.… The general principle to be followed … is to provide the reader not only with information about statistical significance but also with enough information to assess the magnitude of the observed effect or relationship. (pp. 25–26)

Lesson 4 Inference for numerical variables

Let us take an example of acceptance of workplace bullying across geographical regions of the world

What we are interested to do in the slides ahead is :

Compare two means. The means are for different regions in our example. We are trying to see here if the response variable ( acceptance level) is affected by the region (explanatory variable).
Working with small samples
Bootstrapping
Comparing many means

Hypothesis testing for paired data

Let us take the case of reading and writing scores of several high school students. The two scores are not independent.

Here the parameter of interest is the mean difference between the read and write scores for all high school students, μ_diff, and the point estimate of this is the mean difference for the sampled students, x'_diff.
If there were no difference between the reading and writing scores of the population on the average, we would expect a μ_diff of 0. And this is the Null Hypothesis. The alternate Hypothesis is that μ_diff not equal to 0. So we need a two-sided p-value. Note that our sample data is simply the diff column seen above.

Confidence Interval for paired data

Comparing Independent Means

Take the following example. We want to know if highest degree is correlated with hours spent at work. Since the two are not paired variables (i.e. given a person, his working hours are not dependent in the sense that the working hours will still vary randomly even if the highest degree is something), here it makes more sense to compare the means of the working hours per highest degree. For simplicity we bunch the highest degree into simply two categories - college degree and no college degree.

Note here, that the way SE (standard error ) is calculated is different from before.

Bootstrapping

When your sample size is small, and/or sample distribution is skewed, we cannot directly apply the above procedures of HT and CI , hence we literally generate data points. This is called bootstrapping.

T-distribution

T-statistic - Example

Confidence Intervals

Now we compare means for two small samples using T-statistic, for the same data set as above

Note the standard error calculation for difference of two sample means. The standard error is the sum (not difference) of the standard deviation contributions.

Comparing more than two means (ANOVA)

Variability Partitioning

Conditions for ANOVA

ANOVA - Multiple comparisons

While a low value of F-statistic does tell us that at least one pair of means are different, how do we know which groups have different means. Note that for g groups we have g(g-1)/2 pairs of groups, and therefore the same number of pairs of means to compare.
With such a large number of comparisons it is good to be conservative while rejecting the null hypothesis hence we use a corrected significance level (α )

α* = α / K where K=g(g-1)/2 is the number of comparisons -- This is called Bonferroni correction

Lesson 5 - Inference for categorical variables

Here the sample statistic is not the sample mean but the proportion of members of the sample in a particular category. We assume binary(yes/no) valued categorical variables first.

As before, the sampling distribution of the proportion is nearly normal around the true (population) proportion. Note the conditions below for the CLT to hold.

Hypothesis test for a proportion

Estimating the difference between two proportions

Hypothesis tests for comparing two proportions

Note that the pooled proportion is used for Hypothesis (the null hypothesis is 'there is no difference between the population proportions for Male and Female' in the example below) testing

Small Sample proportion

The sample size is such that the Success-Failure condition is not satisfied. Then what we do is that we assume the null hypothesis (for example p=0.5) and do a simulation of the sample with say, coin tosses. For each trial we find the sample statistic ( i.e. the proportion of successes) and plot them on a dot plot. Then we find the percentage of dots that have at least as much z-score as the observed proportion. If this is less than 5% then we can reject the null hypothesis with 95% confidence.