reference from Matrix Algebra for Engineers by Jeffrey R. Chasnov
Inner and outer products
The inner product (or dot product or scalar product) between two vectors is obtained from the ma- trix product of a row vector times a column vector
\[ \mu^{T}\nu = ( \mu_{1} \quad \mu_{2} \quad \mu_{3} ) \begin{pmatrix} \nu_{1} \\ \nu_{2} \\ \nu_{3} \end{pmatrix} = \mu_{1}\nu_{1} + \mu_{2}\nu_{2} + \mu_{3}\nu_{3}. \]
If the inner product between two vectors is zero, we say that the vectors are orthogonal.
The norm of a vector is defined by
\[ \lVert \mu \rVert = (\mu^{T}\mu) = ( \mu_{1}^{2} + \mu_{2}^{2} + \mu_{3}^{2} )^{\frac{1}{2}} \]
If the norm of a vector is equal to one, we say that the vector is normalized. If a set of vectors are mutually orthogonal and normalized, we say that these vectors are orthonormal.
An outer product is also defined, and is used in some applications. The outer product between u and v is given by
\[ \mu\nu^{T} = \begin{pmatrix} \mu_{1} \\ \mu_{2} \\ \mu_{3} \end{pmatrix} ( \nu_{1} \quad \nu_{2} \quad \nu_{3} ) = \begin{pmatrix} \mu_{1}\nu_{1} & \mu_{1}\nu_{2} & \mu_{1}\nu_{3} \\ \mu_{2}\nu_{1} & \mu_{2}\nu_{2} & \mu_{2}\nu_{3} \\ \mu_{3}\nu_{1} & \mu_{3}\nu_{2} & \mu_{3}\nu_{3} \\ \end{pmatrix} \]
Inverse matrix
Square matrices may have inverses. When a matrix A has an inverse, we say it is invertible and denote its inverse by \(A^{−1}\). The inverse matrix satisfies
\[AA^{−1} = A^{−1}A = I.\]
\[A^{-1} = \frac{1}{ad-bc} \begin{pmatrix} d & -b \\ -c & a \\ \end{pmatrix} \]
\[ det \ A = \begin{vmatrix} a & b \\ c & d \\ \end{vmatrix} = ad - bc \]
The determinant of a two-by-two matrix is the product of the diagonals minus the product of the off-diagonals. Evidently, A is invertible only if \(det \quad A \neq 0.\) Notice that the inverse of a two-by-two matrix, in words, is found by switching the diagonal elements of the matrix, negating the off-diagonal elements, and dividing by the determinant. Later, we will show that an n-by-n matrix is invertible if and only if its determinant is nonzero. This will require a more general definition of determinant.
Orthogonal matrices
A square matrix Q with real entries that satisfies
\[Q^{−1} = Q^{T}\]
is called an orthogonal matrix.
Since the columns of \(Q^{T}\) are just the rows of Q, and \(QQ^{T}\) = I, the row vectors that form Q must be orthonormal. Similarly, since the rows of \(Q^{T}\) are just the columns of Q, and \(Q^{T}Q\) = I, the column vectors that form Q must also be orthonormal.
Rotation matrices
Writing the equations for \(x'\) and \(y'\) in matrix form, we have
\[ \begin{pmatrix} x' \\ y' \\ \end{pmatrix} = \begin{pmatrix} cos\theta & -sin\theta \\ sin\theta & cos\theta \\ \end{pmatrix} = \begin{pmatrix} x \\ y \\ \end{pmatrix} \]
The above two-by-two matrix is a rotation matrix and we will denote it by \(R_{θ}\). Observe that the rows and columns of \(R_{θ}\) are orthonormal and that the inverse of \(R_{θ}\) is just its transpose. The inverse of \(R_{θ}\) rotates a vector by −θ.
Permutation matrices
Another type of orthogonal matrix is a permutation matrix. An n-by-n permutation matrix, when multiplying on the left permutes the rows of a matrix, and when multiplying on the right permutes the columns. Clearly, permuting the rows of a column vector will not change its norm.
\[ \begin{pmatrix} 0 & 1 \\ 1 & 0 \\ \end{pmatrix} \begin{pmatrix} a & b \\ c & d \\ \end{pmatrix} = \begin{pmatrix} c & d \\ a & b \\ \end{pmatrix} \]
\[ \begin{pmatrix} a & b \\ c & d \\ \end{pmatrix} \begin{pmatrix} 0 & 1 \\ 1 & 0 \\ \end{pmatrix} = \begin{pmatrix} b & a \\ d & c \\ \end{pmatrix} \]
Gaussian elimination
\[ \begin{align*} −3x_{1} + 2x_{2} − x_{3} = −1 \\ 6x_{1} − 6x_{2} + 7x_{3} = −7, \\ 3x_{1} − 4x_{2} + 4x_{3} = −6, \\ \end{align*} \]
which can be written in matrix form as
\[ \begin{pmatrix} -3 & 2 & -1 & -1 \\ 6 & -6 & 7 & -7 \\ 3 & -4 & 4 & -6 \\ \end{pmatrix} \]
Row reduction is then performed on this augmented matrix. Allowed operations are (1) interchange the order of any rows, (2) multiply any row by a constant, (3) add a multiple of one row to another row. These three operations do not change the solution of the original equations. The goal here is to convert the matrix A into upper-triangular form, and then use this form to quickly solve for the unknowns x.
We first form what is called an augmented matrix by combining the matrix A with the column vector b
\[ \begin{pmatrix} -3 & 2 & -1 & -1 \\ 0 & -2 & 5 & -9 \\ 0 & -2 & 3 & -7 \\ \end{pmatrix} \]
These equations can be solved by back substitution, starting from the last equation and working backwards. We have
\[ \begin{align*} & x_{3} = −1, \\ & x_{2} = −\frac{1}{2}(−9−5x_{3})=2, \\ & x_{1} = −\frac{1}{3}(−1+x_{3}−2x_{2})=2 \end{align*} \]
We have thus found the solution
\[ \begin{pmatrix} x_{1} \\ x_{2} \\ x_{3} \\ \end{pmatrix} = \begin{pmatrix} 2 \\ 2 \\ -1 \\ \end{pmatrix} \]
When performing Gaussian elimination, the diagonal element that is used during the elimination procedure is called the pivot
Reduced row echelon form
If we continue the row elimination procedure so that all the pivots are one, and all the entries above and below the pivots are eliminated, then we say that the resulting matrix is in reduced row echelon form
\[ \begin{pmatrix} 1 & 2 & 3 & 4 \\ 4 & 5 & 6 & 7 \\ 6 & 7 & 8 & 9 \\ \end{pmatrix} \\ \] \[ rref(A) = \begin{pmatrix} 1 & 0 & -1 & -2 \\ 0 & 1 & 2 & 3 \\ 0 & 0 & 0 & 0 \\ \end{pmatrix} \]
Computing inverses
\[(A | E) \rightarrow (E | A^{-1})\]
Because: \[\begin{align} & p_{1}p_{2}p_{3}p_{4}...p_{n}A \rightarrow E \\ & p_{1}p_{2}p_{3}p_{4}...p_{n} = A^{-1} \\ & p_{1}p_{2}p_{3}p_{4}...p_{n}E \rightarrow A^{-1} \end{align}\]
\[ \begin{pmatrix} -3 & -2 & -1 & 1 & 0 & 0 \\ 6 & -6 & 7 & 0 & 1 & 0\\ 3 & -4 & 4 & 0 & 0 & 1 \\ \end{pmatrix}\rightarrow \begin{pmatrix} 1 & 0 & 0 & -\frac{1}{3} & \frac{1}{3} & -\frac{2}{3} \\ 0 & 1 & 0 & \frac{1}{4} & \frac{3}{4} & -\frac{5}{4} \\ 0 & 0 & 1 & \frac{1}{2} & \frac{1}{2} & -\frac{1}{2} \\ \end{pmatrix} \]
LU decomposition
row reduction of a matrix A can be written as
\[M_{3}M_{2}M_{1}A = U\]
where U is upper triangular. Upon inverting the elementary matrices, we have
\[A = M_{3}^{-1}M_{2}^{-1}M_{1}^{-1}U\]
Therefore,
\[L = M_{3}^{-1}M_{2}^{-1}M_{1}^{-1}\]
Our LU decomposition of A is therefore:
\[ \begin{pmatrix} -3 & -2 & -1\\ 6 & -6 & 7\\ 3 & -4 & 4\\ \end{pmatrix} = \begin{pmatrix} 1 & 0 & 0\\ -2 & 1 & 0\\ -1 & 1 & 1\\ \end{pmatrix} \begin{pmatrix} -3 & 2 & -1\\ 0 & -2 & 5\\ 0 & 0 & -2\\ \end{pmatrix} \]
Solving (LU)x = b
The LU decomposition is useful when one needs to solve Ax = b for many right-hand-sides. With the LU decomposition in hand, one writes
\[(LU)x = L(Ux) = b\]