MATH3821 Statisical Modelling and Computing, UNSW
Term 2, 2023
In your report please include all relevant R commands and outputs directly relevant to your answers. The
total marks for this assignment is 35. Up to 3 marks will be given to the overall presentation, i.e., clarity and
conciseness. The assignment is due on Monday June 19 at 5pm and should be submitted through the
link titled “Submission link - Assignment 1” on the subject’s moodle page.
Question 1 [5 Marks]
Use the command data<-read.csv("MATH3821_A#.csv") to import the data MATH3821_A#.csv to R, where
# is the last digit of your student ID (first download the data from the subject’s moodle page and set the
working directory of R to the folder where the data is stored, which can be done through the “Session”
tab). Construct pairwise plots of the data. Does a linear model appear to be appropriate? Comment on
whether there are apparent relationships between the predictors (the x’s) and response variable (y), and
which variable/s you expect to be significant as predictors.
Question 2 [10 marks]
Consider the linear model
yi = β0 +
4∑
j=1
βjxij + εi,
where the εi are independent normal mean zero errors, with common variance σ2.
(a) Write down an expression for the linear model, using the notation Y as the response vector, X as the
design matrix, β as the vector of coefficents, and ε as the vector of errors.
(b) Write down the expression for the log likelihood function. Use this expression to derive the maximum
likelihood estimator (MLE) for β and show that the MLE is equivalent to the least squares estimator
for β:
b = argmin
β
n∑
i=1
(
yi − β0 −
4∑
j=1
βjxij
)2
.
(c) Show that the MLE b is unbiased and derive its variance.
(d) Use your answer to (b) to compute b for the data in Question 1. Include the code you used for
this computation. (You may wish to use the commands X <- cbind(1,as.matrix(data[,-1])) and
Y<-as.vector(data[,1])).
Question 3 [5 marks]
Using R’s lm() function, fit the same linear model as in Question 2. Do you think any predictor/s can be
removed? Write down the resulting estimate, b∗, for β once you have removed the predictor/s. Explain how
you came to your estimate by including all R output and referring to any relevant tests used.
1
Question 4 [7 marks]
Removing non-significant predictors, as we did in Question 3, is a way of penalising model complexity. It falls
into the category of selection methods for reducing model complexity: we include only the predictors that
contribute significantly to the fit of the model. In this question we consider a different approach to penalising
model complexity. It is a regularisation method for reducing model complexity: we include all predictors in
the model, but restrict the values the coefficients can take through a penalty term.
We again fit the model in Question 2. For λ ≥ 0, consider the penalised least squares regression estimator :
b∗∗ = argmin
β

n∑
i=1
(
yi − β0 −
4∑
j=0
βjxij
)2
+ λ
4∑
j=0
β2j
 .
(a) Following arguments similar to Question 2(b) and using the same notation, show that the penalised
least squares regression estimator is given by
b∗∗ = (XTX + λI)−1XTY,
where I is the identity matrix.
(b) Compute the mean and variance of b∗∗. Compare your answer to Question 2(c), what do you notice?
(c) Use your answer to (a) to compute b∗∗ for the data in Question 1 when λ = 1/5. (You may use the X
and Y from Question 2 and the command I<-diag(5).)
Question 5 [5 marks]
In this question we will compare the models we have fit in Question 2–4 using the PRESS (predicted residual
error sum of squares) statistic (Lecture 3, Slide 28). Provided X and Y are properly defined, the following
code can be used to compute the PRESS statistic for the model obtained in Question 2.
PRESS<-0
for(i in 1:length(Y)){
Y.i<-Y[-i]
X.i<-X[-i,]
beta.i<-solve(t(X.i)%*%X.i)%*%t(X.i)%*%Y.i
PRESS<-PRESS+(Y[i]-X[i,]%*%beta.i)ˆ2
}
PRESS
(a) By modifying the above code or otherwise, compute the PRESS statistic (using the data in Question 1)
for the models obtained in Questions 2, 3, and 4 (with λ = 1/5), and comment on the values you obtain.
(b) In one or two sentences, suggest a method for choosing the value of λ for the penalised least squares
regression estimator in Question 4.
(c) Which of the models obtained in Questions 2, 3, and 4 minimises the residual sum of squares? Explain
your answer in one sentence.