More Statistics - Linear Regression

For part 1 - follow link to basic descriptive statistics 

Lesson 6 - Linear Regression

• Correlation describes the strength of linear association between two variables. Correlation coefficient is denoted as R.
• The magnitude of R is the strength of the correlation while the sign is the direction of association. 
• -1.0 ≤ R ≤ 1.0 . R=0 indicates no correlation, a negative R a negative correlation and positive R, a positive correlation.
• R is unitless and not affected by changes in the centre or scale of either variable
• The correlation of X with Y is the same as that of Y with X
• R is very sensitive to outliers
• The covariance of (X,Y)$XY$ is defined by

             cov(X,Y)=𝔼([X−𝔼(X)][Y−𝔼(Y)])

and, assuming the variances are positive, the correlation of 

(X,Y)$XY$ is defined by

R = cor(X,Y)=cov(X,Y)σ(X)σ(Y)

• Residuals

Are leftovers from the model fit.

• Least Squares Line

Is a line that minimises the sum of squares of residuals. Why squares and not say, absolute value ? Because squares ensure that if a residual is twice that another, then its square is more than twice ( 4 times) that of the square of the other residual. This is also the reason why an outlier can pull the line towards it ( so as to reduce the square of its  large residual contribution).

• Prediction and extrapolation

Using the linear model to predict the y-value (response variable value) for a x-value (explanatory variable value)  is prediction. Just plugin the value of x in the linear model equation.

Extrapolation is applying the model to values outside the realm of the original data.This can be dangerous and inaccurate sometimes.

• Conditions for Linear regression

1. Linearity
2. nearly normal residuals
3. constant variability

• R^2

• Regression with categorical explanatory variables

Suppose X is a categorical variable with 5 levels, then the linear model will be of the form

y = b0 + b1.x1 + b2.x2 + b3.x3 + b4.x4 

where X can take 5 possible vector values - (1,0,0,0,0), (0,1,0,0,0), (0,0,1,0,0), (0,0,0,1,0), (0,0,0,0,1)

• Outliers

Note influential points may increase R^2 falsely implying strong correlation. Hence always inspect the plots for outliers. Outliers may be valuable indicators of insufficiency or inappropriateness of our  model, or of two regions which need to be modelled separately.

• Inference for linear regression

1. Hypothesis testing for significance of predictor (slope and intercept)
2. confidence interval for slope
3. conditions for inference

• Testing for the slope - hypotheses

                    H0: β =0 (Nothing is happening. The explanatory variable is not a significant predictor. slope is 0)

                    Ha :β ≠0 (Explanatory variable is a significant predictor of the response variable. slope is non zero)

• Testing for the slope - mechanics

The T-statistic for the slope is give by :

                   T = point estimate - null value   =  b1 - 0

                         ---------------------------------     --------   

                          SE                                             SE_b1

                   where  SE_b1 is the standard error of the point estimate

        degrees of freedom are :

                 df = n-2  ( we lose a degree of freedom for each parameter estimated. we estimate the slope and intercept here)

And the rest of the procedure for significance levels and hypothesis testing follows as usual.

•            Variability Partitioning to check the strength of linear regression model

How much of the variability in Y is explained by X, and how much is unexplained. If the former is higher than the latter then there is a strong correlation and the linear model is strong. This requires ANOVA to partition the variability in Y into explained (due to X) and unexplained.

Lesson 7 - Multiple Linear Regression

Multiple explanatory variables predict a response variable. We are only going to see the case of numerical response variables and numerical/categorical explanatory variables.

Example:

• Multiple predictors

• Inference for MLR
• What are the significant predictors
• HT and CI for the slope
• interpretations

Note that we got all the Std. errors, F-statistics etc. from the R call itself

• How to select a model for prediction of response variable

What we want to know is how to choose explanatory variables. We want only as many of them as contribute significantly to the variability of the response variable. This is the principle of parsimony.

• Adjusted R^2
• Calculation
• Uses

• Model selection
• Stepwise model selection
• p-value and adjusted R^2
• expert opinion

• MLR diagnostics

For part 1 - follow link to basic descriptive statistics