University of Toronto
Department of Statistical Sciences
STA437/2005 Methods of Multivariate Data
Handout: Supplementary Material on Linear Algebra September 2022
ˆ Let x′ = [x1, x2, . . . , xn] and y′ = [y1, y2, . . . , yn] be two real vectors, each with n components.
In the slides, we proved that the projection of x on y is given by
x′y
y′y
y
We can obtain this result in a different and a more straightforward way. Let us denote the
projection of x on y by p. Clearly, p = cy for some constant c (because the projection is in the
direction of y). We next note that the vectors y and x− p are orthogonal; therefore
⟨x− cy,y⟩ = 0 ⇒ c = x
′y
y′y
We conclude that the projection is
p = cy =
x′y
y′y
y =
y′x
y′y
y (1)
or equivalently
p =
⟨x,y⟩
⟨y,y⟩y
The projection can also be represented via matrix multiplication. Let p = P x, where P is the
required projection matrix. From (1)
P =
yy′
y′y
(2)
Here, y is an n× 1 vector, therefore yy′ is an n× n matrix; but y′y is a real number.
ˆ Orthogonal projection matrices are symmetric and idempotent. Let us prove that P in (2)
satisfies these properties. Symmetry is obvious (because yy′ is symmetric), and
P 2 =
yy′
y′y
· yy
′
y′y
=
1
(y′y)2
yy′yy′
=
yy′
y′y
= P
which completes the proof.
ˆ Let A ∈ Rk×k. The trace of matrix A is the sum of its diagonal elements, i.e.,
tr(A) =
∑
1≤i≤k
aii
1
Two important properties of tr(·) are as follows:
1. If A ∈ Rn×k and B ∈ Rk×n, then tr(AB) = tr(BA).
Proof.
tr(AB) =
∑
1≤i≤n
∑
1≤j≤k
aijbji
=
∑
1≤j≤k
∑
1≤i≤n
bjiaij
= tr(BA)
2. If A,P ∈ Rk×k and P is invertible, then tr(P −1AP ) = tr(A)
Proof.
tr(P −1AP ) = tr(APP −1)
= tr(A)
where in the first step, we used Property 1, i.e., tr(AB) = tr(BA).
ˆ We call matrices A,B ∈ Rk×k similar if
B = P −1AP
for some invertible matrix P ∈ Rk×k.
Similar matrices share many interesting properties, e.g., we just proved that they have the
same trace. Similar matrices have the same eigenvalues.
Proof. We show that A and B have the same characteristic polynomials
|P −1AP − λI | = |P −1AP − λP −1P |
= |P −1||A − λI ||P |
= |A − λI |
Since the trace is the sum of the eigenvalues, this also proves that similar matrices have the
same trace.
However, starting from Ae = λe, we have
APP −1e = λe
which gives
P −1APP −1e = λP −1e
but B = P −1AP , thus
BP −1e = λP −1e
So, two similar matrices have the same eigenvalues, but not necessarily the same eigenvectors.
Indeed, we just proved that the eigenvectors of B are given by P −1e.
We did not discuss all the properties of similar matrices in this short note. Similar matrices
have the same determinant, and on the space of all square matrices, similarity is an equivalence
relation. For more information, look at the suggested references on linear algebra.
2
ˆ Let A ∈ Rk×k be symmetric. We proved in the slides that A has real eigenvalues. We also
showed that the eigenvectors of distinct eigenvalues are orthogonal. Moreover, using the spec-
tral decomposition of symmetric matrices
A = PΛP ′
where P has the eigenvectors of A as columns and Λ is a diagonal matrix whose diagonal
elements are the corresponding eigenvalues.
As an example, let us consider the quadratic form Q(x) = 3x21 + 2x1x2 + 3x
2
2, which can be
written as
Q(x) = [x1, x2]
[
3 1
1 3
] [
x1
x2
]
= X ′AX
with A =
[
3 1
1 3
]
.
It is easy to verify that the eigenvalues of A are λ1 = 2 and λ2 = 4, and the corresponding
eigenvectors are e′1 = [1/
√
2,−1/√2] and e′2 = [1/
√
2, 1/
√
2]. Therefore
P =

1√
2
1√
2
− 1√
2
1√
2
 Λ =
[
2 0
0 4
]
As an exercise, verify that P ′AP is equal to Λ.
Now, let X = P Y , i.e., [
x1
x2
]
=

1√
2
1√
2
− 1√
2
1√
2

[
y1
y2
]
(3)
After substituting x1, x2 from (3) in Q, and after a little rearranging, we obtain Q = 2y
2
1+4y
2
2.
Indeed
Q = X ′AX = X ′PΛP ′X
Since Y = P ′X, the quadratic form can also be written as
Q = Y ′ΛY =
∑
1≤i≤k
λiy
2
i
which is exactly what we noticed in the above example.
ˆ A symmetric matrix A ∈ Rk×k is called positive definite iff every eigenvalue of A is positive.
In the above example, A is positive definite (the eigenvalues are 2 and 4). If A and B are both
positive definite, then the sum A +B is also positive definite.
Every positive definite matrix A is invertible, and A−1 is positive definite too. Indeed, if
A = PΛP ′
then
A−1 = PΛ−1P ′
It is easy to verify that AA−1 = A−1A = I k, where I k denotes the identity matrix of size k×k.
(Recall that P is an orthogonal matrix, i.e., PP ′ = P ′P = I k.)
3