Matrix Decomposition and its Application in Statistics Nishith Kumar Lecturer Department of Statistics Begum Rokeya University, Rangpur. Email nk.!ru"#$gmail.com 1
%vervie& ' ' ' ' ' ' ' '
(ntro)uction LU )ecomposition *R )ecomposition +holesky )ecomposition or)an Decomposition Spectral )ecomposition Singular value )ecomposition -pplications 2
Introduction /his Lecture covers relevant matri0 )ecompositions, !asic numerical metho)s, its computation an) some of its applications. Decompositions provi)e a numerically sta!le &ay to solve a system of linear euations, as sho&n alrea)y in 12ampler, 3#4"5, an) to invert a matri0. -))itionally, -))itionally, they provi)e an important tool for analy6ing the numerical sta!ility of a system. Some of most freuently use) )ecompositions are the LU, QR , Cholesky , Jordan, Spectral decomposition an) Singular alue decompositions . 3
Easy to solve system (Cont.) Some linear system that can be easily solved
The solution:
b3 7 a33 b 7 a 8 88 b 7 a n nn
4
Easy to solve system (Cont.) Lower triangular matrix:
Solution: This system is solved using forward substitution
5
Easy to solve system (Cont.) Upper Triangular Matrix:
Solution: This system is solved using Backward substitution
LU Decomposition LU )ecomposition &as originally )erive) as a )ecomposition of ua)ratic an) !ilinear forms. Lagrange, in the very first paper in his collecte) &orks9 34:#; )erives the algorithm &e call that is use) to solve the system of linear euation. Let A !e a m × m &ith nonsingular suare matri0. /hen there e0ists t&o matrices L an) U such that, &here L is a lo&er triangular matri0 an) U is an upper triangular matri0. u33 u38 " u 88 U = " "
u3m
u mm u8m
2here, A = LU
an)
l 33 l 83 L = l m3
"
l 88
l m 8
" l mm "
?L Lagrange 934@A 3>3@;
-. C. /uring 93#38?3#:=; !
o& to )ecompose -LUF -LUF 9upper triangular; A ≈… ≈U 9upper E k ⋅⋅⋅ E 3 A ⇒ U E 3;−3 ⋅⋅⋅ 9 E E k;−3 U ⇒ A 9 E
− 8 8 − > A No&, − 3@ 8 8 3 " " A − 8 8 8 = − 8 3 " 38 − > A 3 − 3 7 8 " 3 @ − 3@ 8 " " A − 8 8 A − 8 8 3 " " 3 ⇒ " − = 8 = " 3 " − 8 3 " 38 − > A " " − : " − @ 3 − 3 7 8 " 3 @ − 3@ 8
A A = 38 @ A − 8 " − = " − 38
U
E 2
E 1
A
(f each such elementary matri0 E i is a lo&er triangular matrices, it can !e prove) that 9 E 3;−3, ⋅⋅⋅, 9 E E k;−3 are lo&er triangular, an) E 3;−3 ⋅⋅⋅ 9 E E k;−3 is a lo&er triangular matri0. 9 E E 3;−3 ⋅⋅⋅ 9 E E k;−3 then A=LU. Let L=9 E "
Calculation o! L o! L and and U (cont.) A −8 A = 38 − > @ − 3@
8
$ 8
A
3 " "
"
" A
3
3 @
"
" 38
−8 −> − 3@
8
A
8
No& re)ucing the first column &e & e have − 8 8 " " A A − 8 8 3 " − = 8 = − 8 3 " 38 − > A " − 38 3 − 3 7 8 " 3 @ − 3@ 8 " " A −8 A − 8 8 3 " " 3 ⇒ " − = 8 = " 3 " − 8 3 " 38 − > " " − : " − @ 3 − 3 7 8 " 3 @ − 3@
8
8
A
#
Calculation o! L o! L and and U (cont.) No& 3 −8 − 37 8
" "
3
3 " "
−3
3 " " 3 " − @
"
−3
3 "
3 = 8 3 7 8
" " 3 " "
" "
3 " " 3
3 "
"
3 "
@
3 " = 8 3 3 7 8
@
3
/herefore, A −8 A = 38 − > @ − 3@
3 " " A − 8 8 " − = 8 A $ 8 3 " =LU 8 3 7 8 @ 3 " " − : 8
(f - is a Non singular matri0 then for each L 9lo&er triangular matri0; the upper triangular matri0 is uniue !ut an LU )ecomposition )ecomposition is not uniue. /here can !e more than one such LU )ecomposition )ecomposition for a matri0. Such as A A = 38 @
A " " 3 − 8 7 A 8 7 A 8 A $ 38 3 " " − = 8 @ @ 3 " 3@ 8 " :
−8 −>
=LU
1%
alculation L and U (cont.) (cont.) Calculationof o! L o! L and and U /hus LU )ecomposition is not uniue. Since &e compute LU )ecomposition !y elementary transformation so if &e change L then U &ill !e change) such that -LU /o fin) out the uniue LU )ecomposition, )ecomposition, it is necessary to put some restriction on L an) U matrices. matrices. Gor e0ample, &e can reuire the lo&er triangular matri0 L to !e a unit one 9i.e. set all the entries of its main )iagonal to ones;. LU Decomposition in R"
' li!rary9Catri0; ' 0H?matri09c9@,8,3, #,@,=,=,8,: ;,ncol@,nro&@; ' e0pan)9lu90;;
11
alculation of L and U (cont.) '
#ote there are also generali6ations of LU to non?suare an) singular
matrices, such as rank revealing LU factori6ation. ' 1Ian, +./. 98""";. %n the e0istence an) computation of rank revealing LU factori6ations. Linear Algebra and its Applications, @3A 3##?888. ' Ciranian, L. an) :" 3#3A;
Girstly *R )ecomposition originate) &ith >@;. Later Erhar) Schmi)t 93#"4; prove) the *R Decomposition /heorem
(rhard Schmidt
93>4A?3#:#;
(f A is a m×n matri0 &ith linearly in)epen)ent columns, then A can !e )ecompose) as , A = ! &here * is a m×n matri0 &hose columns form an orthonormal !asis for the column space of A an) R is an nonsingular upper triangular matri0. 1
*R?Decomposition 9+ont.; /heorem (f A is a m×n matri0 &ith linearly in)epen)ent columns, then A can !e )ecompose) as , A = ! &here * is a m×n matri0 &hose columns form an orthonormal !asis for the column space of A an) R is an nonsingular upper triangular matri0. Iroof Suppose -1u3 u8 . . . un5 an) rank 9 A; n. -pply the 8#. CAUC;?/)>-=2
Let A !e a m × m real symmetric matri0. /hen there e0ists an orthogonal matri0 9 such that 9 A9 = Λ or A = 9 Λ 9 , &here : is a )iagonal matri0. (
(
34
Spectral Decomposition and &rincipal component Analysis 0Cont12 By using spectral )ecomposition &e can &rite
A = 9 Λ 9 (
(n multivariate analysis our )ata is a matri0. Suppose our )ata is ) matri0. Suppose is mean centere) i.e., ) → 9 ) − µ ; an) the variance covariance matri0 is V. /he variance covariance matri0 V is real an) symmetric. Using spectral )ecomposition &e can &rite ;=9:9 ( . 2here : is a )iagonal matri0. Λ = diag 9λ 3, λ 8 ,, λ ; -lso λ ≥ λ ≥ ≥ λ n
3
8
n
tr9;; /otal variation of Data tr9 :; 35
Spectral Decomposition and &rincipal component Analysis 0Cont12 /he Irincipal component transformation is the transformation 9?W; 9 2here, E9i;" X9i;i +ov9i , $;" if i Y $ X93; Z X9 8; Z . . . Z X9n; n
∑ = r .r 3 7 8 !r (
9 3
(
= 2 , m
(
9 8
=
2 n
= U Λ" appro0imation "
Lo& rank -ppro0imation to flo&ers image
Rank?3"" appro0imation
Rank?38" appro0imation #
Lo& rank -ppro0imation to flo&ers image
an>15% a**ro?imation
/rue (mage
!%
Butlier Detection Using S9D Nishith an) Nasser 98""4,CSc. /hesis; propose a graphical metho) of outliers )etection using SXD. (t is suita!le for !oth general multivariate )ata an) regression )ata. Gor this &e construct the scatter plots of first t&o I+[s, an) first I+ an) thir) I+. 2e also make a !o0 in the scatter plot &hose range lies median93 st9D ; K @ × mad 93 st9D ; in the ) ?a0is an) median98nd9D @rd9D ; K @ × mad 98nd9D @rd9D ; in the Y? a0is. 2here mad me)ian a!solute )eviation. /he points that are outsi)e the !o0 can !e consi)ere) as e0treme outliers. /he points outsi)e one si)e of the !o0 is terme) as outliers. -long &ith the !o0 &e may construct another smaller !o0 !oun)e) !y 8.:78 C-D line !1
Butlier Detection Using S9D 0Cont12 -2K(NS?BR-DU?K-SS 93#>=; D-/Data set containing 4: o!servations &ith 3= influential o!servations. -mong them there are ten high leverage outliers 9cases 3?3"; an) for high leverage points 9cases 33?3=; ?(mon 98"":;. Scatter plot of a&kins, Bra)u an) kass )ata 9a; scatter plot of first t&o I+[s an) 9!; scatter plot of first an) thir) I+. !2
Butlier Detection Using S9D 0Cont12 C%D(G(ED BR%2N D-/Data set given !y Bro&n 93#>";. Ryan 93##4; pointe) out that the original )ata on the :@ patients &hich contains 3 outlier 9o!servation num!er 8=;.
Scatter plot of mo)ifie) Bro&n )ata 9a; scatter plot of first t&o I+[s an) 9!; scatter plot of first an) thir) I+.
(mon an) a)i98"":; mo)ifie) this )ata set !y putting t&o more outliers as cases := an) ::. -lso they sho&e) that o!servation 8=, := an) :: are outliers !y using generali6e) stan)ar)i6e) Iearson resi)ual 9. Daily mean &in) spee) 92S; m7sec #. ours of !right sunshine as percentage of ma0imum possi!le sunshine hours 9CIS; 3". Solar ra)iation 9SR; cal7cm87)ay
!!
+onseuences of SXD 21 &at'ix Alo'itm*5 9ol )1 Easic Decompositions5 Siam5 &hiladelphia1
Catri0 Decomposition. http77fe)c.&i&i.hu?!erlin.)e70plore7e!ooks7html7csa7no)
[email protected]
"1
