Assignment 1 (5%)
Submission: 23:59 on 17th Feb 2023 (Friday of Week 6)
In this assignment, we are interested in using Python to solving a weighted least squares (WLS) problem.
Compared to ordinary least squares that minimizes the mean squared error, the WLS problem instead assigns
a unique weight αi to each sample and minimizes the weighted mean squared error
WMSE(w) =
1
m
m∑
i=1
αi(x
⊤
i w − yi)2,
where (xi, yi) (for 1 ≤ i ≤ m) represents a training sample and its target, αi ∈ R (usually positive) is the
weight assigned to sample i and w = [w1, w2, . . . , wd]
⊤ ∈ Rd are the parameters we want to estimate. As
in lecture, we can stack the training samples and targets into a matrix (known as the design matrix) and
vector respectively. We denote these as
X =

x⊤1
x⊤2
...
x⊤m
 and y =

y1
y2
...
ym
 .
For example for d = 2 and m = 5, we may have the design matrix, target vector respectively as
X =

1 4
4 2
5 6
3 −3
9 −10
 and y =

−1
2
1
0
4
 .
We may also transform the weights α1, α2, . . . , αm into a diagonal weight matrix A ∈ Rm×m as
A =

α1 0 . . . 0
0 α2 . . . 0
...
...
. . .
...
0 0 . . . αm
 .
For example, if (α1, α2, α3, α4, α5) = (1, 2, 1, 3, 0.5), then
A =

1 0 0 0 0
0 2 0 0 0
0 0 1 0 0
0 0 0 3 0
0 0 0 0 0.5
 .
1
The WLS solution for w = argminwWMSE(w) is known to be
w = (X⊤AX)−1X⊤Ay. (1)
(See the last page for a proof that you do not need to know for the purposes of this assignment.)
Write a Python script to find the WLS solution forw given an arbitrary matrixX ∈ R5×2, a vector y ∈ R5
and a diagonal matrixA ∈ R5×5. Submit your Python code as a function (“def A1 MatricNumber(X,A,y)”)
that takes in X, A and y as inputs and generates (X⊤AX)−1 and w as outputs in a single file with the
filename “A1 StudentMatriculationNumber.py”. Your Python routine should return a matrix (X⊤AX)−1
and the WLS solution vector w (as a numpy array). Hint: you will need “import numpy as” and its matrix
manipulation functions.
Precise Instructions:
1. Please use the python template provided to you. Do not comment out any lines. Remember to re-
name both “A1 StudentMatriculationNumber.py” and “A1 MatricNumber” using your student ma-
triculation number. For example, if your matriculation ID is A1234567R, then you should submit
A1 A1234567R.py that contains the function A1 A1234567R.
2. Please do NOT zip/compress your file. Please do not redefine X,y and A inside your function. The
function will take in inputs X ∈ R5×2, y ∈ R5 and A ∈ R5×5.
3. Please test your code at least once. Python is case sensitive.
4. Note that when we test your code, the matrices that are inputs to your function will be non-singular.
5. Because of the large class size, points will be deducted if instructions are not followed carefully. The
way we would run your code might be something like this:
>> import A1_A1234567R as grading
>> InvXTX, w = grading.A1_A1234567R(X,A,y)
6. The allocation of the total mark (5%) is based on the two outputs: InvXTAX (2%) and w (3%).
Gentle Reminders:
1. Please make sure you replace “StudentMatriculationNumber” and “MatricNumber” with your matric-
ulation number!
2. Date of release: Friday of Week 4
3. Submission: Canvas/EE2211/Assignments/Assignment 1
4. The submission folder in Canvas will be closed on 17th Feb 2023 (Friday of Week 6) at 23:59 sharp!
No extensions will be entertained.
2
The following is OPTIONAL knowledge.
Proof of the WLS solution in (1). First we claim that (m times) the objective function can be written in
matrix form as
m∑
i=1
αi(x
⊤
i w − yi)2 = (Xw − y)⊤A(Xw − y). (2)
This is merely a matter of bookkeeping. Note that
Xw − y =

x⊤1
x⊤2
...
x⊤m
w −

y1
y2
...
ym
 =

x⊤1 w − y1
x⊤2 w − y2
...
x⊤mw − ym
 .
As such, the RHS of (2) is
(Xw − y)⊤A(Xw − y) =

x⊤1 w − y1
x⊤2 w − y2
...
x⊤mw − ym

⊤ 
α1 0 . . . 0
0 α2 . . . 0
...
...
. . .
...
0 0 . . . αm


x⊤1 w − y1
x⊤2 w − y2
...
x⊤mw − ym

=

x⊤1 w − y1
x⊤2 w − y2
...
x⊤mw − ym

⊤ 
α1(x
⊤
1 w − y1)
α2(x
⊤
2 w − y2)
...
αm(x
⊤
mw − ym)
 =
m∑
i=1
αi(x
⊤
i w − yi)2 = LHS of (2).
To minimize f(w) = (Xw − y)⊤A(Xw − y), we note that it can be written as
f(w) = w⊤X⊤AXw − 2w⊤X⊤Ay + y⊤Ay.
Differentiating this with respect to w, we obtain
∇f(w) = 2X⊤AXw − 2X⊤Ay.
Setting this to zero and assuming that (X⊤AX) is invertible, we obtain that the optimal w is
w = (X⊤AX)−1X⊤Ay
as desired.
Remark 1. Under what conditions on X and A is X⊤AX invertible?
3