ES2C7 Eng. Math. & Data Analytics Regression
Three main machine learning techniques
13
ES2C7 Engineering Mathematics & Data Analytics Regression
Regression Classification
Clustering
Regression
14
• Predicting the value of a continuous response variable
from a set of predictor variables
• Examples:
• Predict life expectancy given: {smoking, diet, exercise}
• Predicting the extension of a spring given:
{force, temperature, age}
ES2C7 Engineering Mathematics & Data Analytics Regression
PredictorsResponse variable
Example regression problem
15
• A company has developed a low-cost strain gauge for
measuring the weight of an object
• The resistance of the strain gauge should be proportional
to the weight, however the team have noticed that it
seems to be affected by temperature
• The team performs some experiments at different weights
and temperatures to determine how the the strain gauge
is operating. The true weight is measured by a more
expensive device.
ES2C7 Engineering Mathematics & Data Analytics Regression
Example regression problem
16
ES2C7 Engineering Mathematics & Data Analytics Regression
Experiment no Resistance (Ω) Weight (kg) Temperature (°C)
1 1001 10 -40
2 999.4 11 0
3 1004 12 20
4 998 11 30
5 1003 10 20
17
ES2C7 Engineering Mathematics & Data Analytics Regression
weight
re
sis
ta
nc
e
re
sis
ta
nc
e
temperature
18
ES2C7 Engineering Mathematics & Data Analytics Regression
Example regression problem
19
• Assume that the weight (w), temperature (t) and
resistance (R) are all related by the following model:! = ($ ∗ &) + () ∗ *) + + + ,
Where:
→ $, ), and c are constants from our model
→ , is an error term, accounting for errors in our model
Linear regression is a method of finding a, b, c
ES2C7 Engineering Mathematics & Data Analytics Regression
Predictors:! ∈ ℝ! Response variable! ∈ {1, 2, … ,(}
Regression
20
• Learns a function that maps the values of the n predictor(s) variables to
the response variable:
ES2C7 Engineering Mathematics & Data Analytics Regression
-: ℝ! → ℝ
→ The set of predictors: 1", 1#, … , 1$
→ Corresponding function is estimated from training data (t samples)
→ Response variables: 4", 4#, … , 4$
What does a Data Scientist (you!) do for regression?
21
ES2C7 Engineering Mathematics & Data Analytics Regression
1. Training Dataset with
response variables and
predictors
Age at death Weight (kg) Exercise/week (hrs)
78.5 87 5
54.3 56 3
6.7 16 N/A
90.9 60 none
3. Regression algorithm
2. A candidate
model (function)
* = (- ∗/) + (2 ∗ 3) + 4
• A = Age
• W = Weight
• E = Exercise
• a,b,c parameters of
model
! ∈ ℝ " ∈ ℝ!
t=4, n=2
A Cost Function
22
ES2C7 Engineering Mathematics & Data Analytics Regression
• We have a set of t data
points: (!6, #6). . . (!7, #7)
• Let’s begin by defining how well a
model fits the data:
23
ES2C7 Engineering Mathematics & Data Analytics Regression
A Cost Function • We have a set of t data
points: (!6, #6). . . (!7, #7)
• We have a candidate model:
red line, that maps x -> y
• Let’s begin by defining how well a
model fits the data:
A Cost Function
24
ES2C7 Engineering Mathematics & Data Analytics Regression
!!
8!!
• We have a set of t data
points: (!6, #6). . . (!7, #7)
• We have a candidate model:
red line, that maps x -> y
• Using our candidate model,
we can find the predicted
data points:
(!6, ). . . (!7, )
• Let’s begin by defining how well a
model fits the data:
25
ES2C7 Engineering Mathematics & Data Analytics Regression
A Cost Function
9!
!!
8!!
• We have a set of t data
points: (!6, #6). . . (!7, #7)
• We have a candidate model:
red line, that maps x -> y
• Using our candidate model,
we can find the predicted
data points:
(!6, ). . . (!7, )
• For the i-th data point, the
error between our real y
value and predicted value is: ': = #:-&#:
• Let’s begin by defining how well a
model fits the data:
26
ES2C7 Engineering Mathematics & Data Analytics Regression
A Cost Function
9!
!!
8!!
• For a candidate solution to
the problem (red line) the
sum-of-squared errors is:5 = "$ ∑(4% − 84%)# = "$ ∑(9%)#• Let’s begin by defining how well a model fits the data:
27
ES2C7 Engineering Mathematics & Data Analytics Regression
A Cost Function
9!
!!
8!!
• For a candidate solution to
the problem (red line) the
sum-of-squared errors is:5 = "$ ∑(4% − 84%)# = "$ ∑(9%)#
• Since our function is:&# = )! + +
We can write:5 = 1*;(4% − $1% + ) )#
• Let’s begin by defining how well a
model fits the data:
Matrix Formulation
28
ES2C7 Engineering Mathematics & Data Analytics Regression
→ ! = #$ + &
Assuming t data points ($!, !!). . . ($" , !")
→ !! = #$! + &
→ !# = #$# + &
…
→ !" = #$" + &
29
ES2C7 Engineering Mathematics & Data Analytics Regression
Matrix Formulation
→ ! = #$ + &
Can rewrite as: AP = Y
→ * = $! 1⋮ ⋮$" 1 , - = #& , . = !!⋮!"
30
ES2C7 Engineering Mathematics & Data Analytics Regression
Matrix Formulation
→ AP = Y
→ * = $! 1⋮ ⋮$" 1 , - = #& , . = !!⋮!"
→ / = (*- − .)$(*- − .)
How to minimise E? By setting %&%' = 0!
5 = 1*;(4% − $1% + ) )#
31
• Expression that minimises E:
→ < = (=&=)'"=&>
→ This is the ordinary least squares solution
• Or in MATLAB:
→ P = A\y
• This approach is known as linear regression
• It is the “workhorse” of data-driven models
ES2C7 Engineering Mathematics & Data Analytics Regression
Matrix Formulation
Example
32
ES2C7 Engineering Mathematics & Data Analytics Regression
% data points for X-axis
x = [0:0.1:1.0]';
% create the model that we will
try and estimate
y = 2*x + 3;
% add some noise to y to make it
more difficult realistic
ynoise = y + 0.1*randn(11,1);
% let's look at our data
figure;
plot(x,ynoise,'b+');
axis([0 1 0 5])
Example
33
ES2C7 Engineering Mathematics & Data Analytics Regression
% create matrix A
A = [x ones(11,1)];
% perform least squares using
backslash operator
P = A\ynoise;
% get predicted y values
using fitted model
yfit = A*P;
% plot using red line
hold on;
plot(x,yfit,’r’);
How do we extend this to…
34
• Polynomial functions?
• Multiple inputs?
ES2C7 Engineering Mathematics & Data Analytics Regression
Polynomial fitting
35
→ ! = #$# + &$ + 2
Assuming t data points ($!, !!). . . ($" , !")
→ !! = #$!# + &$! + 2
→ !# = #$## + &$# + 2
…
→ !" = #$"# + &$" + 2
ES2C7 Engineering Mathematics & Data Analytics Regression
36
ES2C7 Engineering Mathematics & Data Analytics Regression
→ ! = #$# + &$ + 2
Can rewrite as: AP = Y
→ * = $!# $! 1⋮ ⋮ ⋮$"# $" 1 , - = #&2 , . = !!⋮!"
Polynomial fitting
Exercise 1: Non-Linear Spring
37
ES2C7 Engineering Mathematics & Data Analytics Regression
• How would we formulate this as a least-
squares problem? AP = Y
• Assume t data points
• Model we want to fit is:
→ 4 = $1( + )1# + +1 + ?
• What is Y, A and P?
MATLAB Code
38
ES2C7 Engineering Mathematics & Data Analytics Regression
% create some random data points
m_data = 1000;
x = rand(m_data,1)*2-1;
% create the model and add noise
y = x-0.3*x.^3 + (rand(m_data,1)-
1)*0.1;
% plot a graph of datapoints (blue)
and label
figure; plot(x,y,'b+');
xlabel('spring extension');
ylabel('Force');
MATLAB Code
39
ES2C7 Engineering Mathematics & Data Analytics Regression
% form A matrix (data matrix)
A = [x.^3 x.^2 x ones(m_data,1) ];
% estimate the parameters using
backslash operator
P = A\y
% obtain predicted y values from
model: yfit
yfit = A*P;
%plot the predicted y values on
the same graph
hold on; plot(x,yfit,'r+');
With more than one input
40
ES2C7 Engineering Mathematics & Data Analytics Regression
Exercise 2
41
ES2C7 Engineering Mathematics & Data Analytics Regression
• How would we formulate this
as a least-squares problem?
AP = Y
• Assume t data points
• Model we want to fit is:
→ @ = $1# + )4# + +
• What are the elements in Y, A
and P?
Exercise 2
42
ES2C7 Engineering Mathematics & Data Analytics Regression
> = @"@#⋮@$
Solution:
= = 1"#1##⋮1$#
4"#4##⋮4$#
11⋮1< =
$)+
• How would we formulate this
as a least-squares problem?
AP = Y
• Assume t data points
• Model we want to fit is:
→ @ = $1# + )4# + +
43
ES2C7 Engineering Mathematics & Data Analytics Regression
(!
)(! MSE = "$ ∑(4% − 84%)#
MAE = "$ ∑$)B(4% − 84%)
Mean squared error
Mean absolute error
SSE = ∑(4% − 84%)#Sum squared errors