CISC 271 Class 16
Review #2: Matrices and Linear Regression
Test #2 will examine material from Class 8 to Class 15 inclusive. This material includes:
diagonalizable matrices; data standardization; projection to a subspace; linear regression; cross-
validation; and elementary use of the singular-value decomposition (SVD).
The main concepts for data standardization include:
• The mean of a vector ~u is written as u¯
• The zero-mean form of ~u is ~m(~u) def= ~u−~1 u¯
• The sample standard deviation of ~a is σ~u def= ‖~m(~u)‖/
√
m− 1
• The standardization of ~u is ~z(~u) def= ~m(~u)/σ~u, which has a mean of zero and a variance of
one, i.e., u¯ = 0 and σ2
~z
= 1
The main concepts for vector projection include:
• Error of projection is ~e = ~c− ~p
• Projection of a vector ~c to a subspace V, which is spanned by the columns of a matrix A, is
~p ≈ ~c where ~p = A~w
• Normal equation to find weights of a projection is ATAwˆ = AT~c
• Projection matrix is P = A[ATA]−1AT
• Special case: if A is singular, find PC from a basis for the column space of A
The main concepts for cross-validation of linear regression include:
• Linear regression A~w ≈ ~c is projection of ~c to column space of A
• Measure error ~e root-mean-square error RMS(A,~c; wˆ) = ‖~e(wˆ)‖/√m
• Validation trains using all the data; not representative of actual regression results
• K-fold cross-validation randomly divides data into training subsets and testing subsets
The main concepts for the SVD include:
• Every real matrix A that has rank r can be factored as A = UΣV where U and V are
orthogonal matrices, Σ is a “diagonal” matrix, and singular values are ordered as σ1 ≥ σ2 ≥
· · · ≥ σr > 0 · · ·0
• Left singular vectors of non-zero singular values are a basis for the column space of A
• Right singular vectors of zero singular values are a basis for the null space of A
• Left singular vectors of zero singular values are a basis for the orthogonal complement of
the column space of A, written as ⊥U
• Right singular vectors of non-zero singular values are a basis for the row space of A
104 c© R E Ellis 2021