Matrix Algebra for Engineers (part two)

reference from Matrix Algebra for Engineers by Jeffrey R. Chasnov

Vector spaces

  • A vector space consists of a set of vectors and a set of scalars.
  • For the set of vectors and scalars to form a vector space, the set of vectors must be closed under vector addition and scalar multiplication. That is, when you multiply any two vectors in the set by real numbers and add them, the resulting vector must still be in the set.

As an example:


so that the set of all three-by-one matrices (together with the set of real numbers) is a vector space. This space is usually called \mathbb{R^{3}}.

Linear independence

The set of vectors, ${u_{1}, u_{2}, . . . , u_{n}}$, are linearly independent if for any scalars $c_{1}, c_{2}, . . . , c_{n}$, the equation

has only the solution $c_{1} = c_{2} = ··· = c_{n} = 0$

Span, basis and dimension


Given a set of vectors, one can generate a vector space by forming all linear combinations of that set of vectors. The span of the set of vectors ${v_{1}, v_{2}, . . . , v_{n}}$ is the vector space consisting of all linear combinations of $v_{1}, v_{2}, . . . , v_{n}$. We say that a set of vectors spans a vector space.


The smallest set of vectors needed to span a vector space forms a basis for that vector space


The number of vectors in a basis gives the dimension of the vector space

Gram-Schmidt process

Given any basis for a vector space, we can use an algorithm called the Gram-Schmidt process to construct an orthonormal basis for that space

Let the vectors $v_{1}, v_{2}, . . . , v_{n}$ be a basis for some n- dimensional vector space. We will assume here that these vectors are column matrices, but this process also applies more generally. We will construct an orthogonal basis $u_{1}, u_{2}, . . . , u_{n}$, and then normalize each vector to obtain an orthonormal basis.

First, define $u_{1} = v_{1}$, To find the next orthogonal basis vector, define

The next orthogonal vector in the new basis can be found from

We can continue in this fashion to construct n orthogonal basis vectors. These vectors can then be normalized via

Null space

The null space of a matrix A, which we denote as Null(A), is the vector space spanned by all column vectors x that satisfy the matrix equation

To find a basis for the null space of a noninvertible matrix, we bring A to reduced row echelon form.

We call the variables associated with the pivot columns, x1 and x3, basic variables, and the variables associated with the non-pivot columns, x2, x4 and x5, free variables. Writing the basic variables on the left-hand side of the $Ax = 0$ equations, we have from the first and second rows

Eliminating x1 and x3, we can write the general solution for vectors in Null(A) as

where the free variables $x_{2}$, $x_{4}$, and $x_{5}$ can take any values. By writing the null space in this form, a basis for Null(A) is made evident, and is given by

Application of the null space

An underdetermined system of linear equations $Ax = b$ with more unknowns than equations may not have a unique solution

If u is the general form of a vector in the null space of A, and v is any vector that satisfies $Av = b$, then $x = u+v$ satisfies $Ax = A(u+v) = Au+Av = 0+b = b$. The general solution of $Ax = b$ can therefore be written as the sum of a general vector in Null(A) and a particular vector that satisfies the underdetermined system.

As an example,

We first bring the augmented matrix to reduced row echelon form:

The null space is determined from $x_{1} = 0$ and $x_{2} = −\frac{x_{3}}{2}$, and we can write

The general solution to the original underdetermined linear system is the sum of the null space and the particular solution and is given by

Column space

The column space of a matrix is the vector space spanned by the columns of the matrix. When a matrix is multiplied by a column vector, the resulting vector is in the column space of the matrix, as
can be seen from

Recall that the dimension of the null space is the number of non-pivot columns—equal to the number of free variables—so that the sum of the dimensions of the null space and the column space is equal to the total number of columns—equal

Row space, left null space and rank

In addition to the column space and the null space, a matrix A has two more vector spaces associated with it, namely the column space and null space of $A^{T}$, which are called the row space and the left null space.

Furthermore, the dimension of the column space of A is also equal to the number of pivot columns,
so that the dimensions of the column space and the row space of a matrix are equal. We have

We call this dimension the rank of the matrix A

Orthogonal projections

Suppose that V is an n-dimensional vector space and W is a p-dimensional subspace of V. In general, the orthogonal projection of v onto W is given by

and we can write

$v_{proj_{W}}$ is closer to v than any other vector in W

The least-squares problem


These equations constitute a system of n equations in the two unknowns $\beta_{0}$ and $\beta_{1}$. The corresponding
matrix equation is given by:

This is an overdetermined system of equations with no solution. The problem of least squares is to find the best solution.

We can generalize this problem as follows. Suppose we are given a matrix equation, $Ax = b$, that has no solution because b is not in the column space of A. So instead we solve $Ax = b_{projCol(A)}$ , where $b_{projCol(A)}$ is the projection of b onto the column space of A. The solution is then called the least-squares solution for x.

Laplace expansion

which can be written suggestively as

Properties of a determinant

  • Property 1: The determinant of the identity matrix is one;
  • Property 2: The determinant changes sign under row interchange;
  • Property 3: The determinant is a linear function of the first row, holding all other rows fixed.

The eigenvalue problem

Let A be a square matrix, x a column vector, and λ a scalar. The eigenvalue problem for A solves

for eigenvalues $\lambda_{i}$ with corresponding eigenvectors $x_{i}$. Making use of the identity matrix I, the eigenvalue problem can be rewritten as

For there to be nonzero eigenvectors, the matrix $(A − \lambda I)$ must be singular, that is,

Matrix diagonalization

For concreteness, consider a two-by-two matrix A with eigenvalues and eigenvectors given by

And consider the matrix product and factorization given by

Generalizing, we define S to be the matrix whose columns are the eigenvectors of A, and Λ to be the diagonal matrix with eigenvalues down the diagonal. Then for any n-by-n matrix with n linearly independent eigenvectors, we have

where S is an invertible matrix. Multiplying both sides on the right or the left by $S^{−1}$, we derive the

Powers of a matrix

Suppose that A is diagonalizable, and consider