Week 2 Lecture 1: Linear Regression

Week 2 Lecture 1: Linear Regression

Regression in X and Y

Firstly for regression Y is continious

X \( \rightarrow \) Y

Linear Regression

Simple Linear Regression:

\( Y = \theta _0 + \theta _ 1 X \)

Obviously, \( \theta _0 \) is the Y axis intercept, while \( \theta _ 1 \) is the population slope.

Multiple Linear Regression

\( Y = \theta _0 + \theta _ 1 X + \theta _ 2 X_2 + … + \theta _ p X_p \)

Thus \( \theta _i \) where i>0 is the slope of the i^th variable X_i, and \( \theta _0 \) is the average value of Y if all X are made 0.

These are the expected values, in actuality, both LR in single and multiple have an extra random error term \( \epsilon \) added at the end.

We try to find \( \theta _0 , \theta _ 1 , \theta _ 2 , … , \theta _ p \) such that it fits the given data closely.

Assumptions in LR

Consider simple LR: \( Y = \theta _0 + \theta _ 1 X + \epsilon \)

Let E(p) denote expectation of P.

The assumption we make are

  1. Dataset is linear
  2. E( \(\epsilon _i \) ) = 0 [ie mean of random errors is 0]
  3. \(\sigma \) ( \(\epsilon _i \) ) = \(\sigma _0 \) [Some std dev exists]
  4. Errors are independent of each other
  5. Errors are normally distributed

This kind of noise is called Gaussian noise

Fitting

We try to find \( \theta _0 , \theta _ 1 , \theta _ 2 , … , \theta _ p \) such that it fits the given data closely, such that the sum of squares of vertical distances between data points and the line is the least [ called Least Squares regression line, and this is a unique line ]

Given training points \(x_i , y_i ) \)

Sum of sq errors is given by \( \Sigma ( y_i - ( \theta _0 + \theta _1 x_1 )^2 \)

And we want to minimise this by changing \( \theta _i \) This can be done by taking pratial derivative of the function wrt the coefficients \( \theta \) and setting them to 0

\( \theta _1 = \frac{n\Sigma xy - \Sigma x \Sigma y}{ n \Sigma x^2 - (\Sigma x)^2} \)

\( \theta _0 = \frac{\Sigma y - \theta _1 \Sigma x }{ n} \)

The reduction can be done using Gradient Descent

Gradient Descent