Linear regression is basically a predictive analytics technique. Regression is used to understand and quantify cause-effect relationships. It helps us to understand relationship between dependent and independent variables. Example:- Simplest case we are trying to find is the relationship between baby and birth weight and gestation period.
Linear relationship equation is usually shown as
Y = B0 +B1X1 + e
where, B0 = Intercept
e = error
To understand the relationship between X & Y , we need to figure out the values of the BETA coefficients.
Beta Coefficient(s):- The estimate of magnitude of impact of changes in the predictor(s) (IV) on the predicted variable (DV).
In the birth weight example, we believe:
Birth weight = B0 + B1 * Gestation Period + e
We now need to estimate what the beta coefficients values are, from the data available to us, that will best capture the relationship between birth weight & gestation period. If we know the value of B0 & B1 , we would be able to know exact relationship between X & Y. Towardsit only our effort are essentially directed at in linear regression.
Ordinary Least Squares Regression (OLS):- It is a technique that estimates coefficients on the variables hypothesized to have an impact on the variable of interest by identifying the line that minimizes the sum of squared differencs between points on the estimated line and the actual values of the independent variable. We are going to use OLS regression technique to estimates coefficients on the variable i.e. B0 & B1
- Many lines are possible that can pass through points, there won’t be single line that captures the all possible point as there is a random as there is a random variation.
- OLS tries to identify best possible line and slope & intercept on the best possible lines are essentially beta coefficients.
- One way of choosing a line among all possible lines is to identify the line that would explain most variation in Y – in other words, have least total error.
- Best line is which is as close as possible to as many points as possible.
- The line that has minimum total sum of square distances has to be the best possible line.
Q = ∑ (Yi-b0-b1x1)²
If we calculate b0 & b1 using the following formula, than we will automatically get the best line possible, this estimates are called Ordinary Least Square Estimates.
Note: Coefficients i.e. B0 & B1 helps us to estimate a straight line that best captures the relationship between birthweight and gestation period.
Once we estimate the coefficients, we have an equation like this (Birth Weight Example):
Birthweight = Intercept estimate + Beta Coefficient * Gestation
Remember this is the best fitted line, but this line will will not cover every single point on the scatter plot.