Consider the case when the matrix $A$ is invertible.
Even if $A$ is not invertible, note that $A-\epsilon I$ is invertible matrix for sufficiently small $\epsilon$.
(See Problem Perturbation of a singular matrix is nonsingular for a proof of this fact.)
Take the limit $\epsilon \to 0$.
Proof.
We want to show that $|AB-\lambda I|=|BA-\lambda I|$, where $\lambda$ is an unknown and $I$ is the $n \times n$ identity matrix.
First let us consider the case when the matrix $A$ is invertible.
So suppose that $A$ is invertible.
Then we have
\begin{align*}
|AB-\lambda I| &=|A^{-1}(AB-\lambda I)A| =|BA-\lambda I|.
\end{align*}
Here we use the fact that $|A||A^{-1}|=|AA^{-1}|=|I|=1$.
Thus when $A$ is invertible, the claim is proved.
Now we consider the general case.
Whether or not $A$ is invertible, the matrix $A-\epsilon I$ is invertible for sufficiently small $\epsilon$.
(More precisely, if $\epsilon$ is smaller than absolute values of all nonzero eigenvalues of $A$. then $A-\epsilon I$ is invertible. See Problem Perturbation of a singular matrix is nonsingular for a proof.)
Then by the previous case, we have
\[ |(A-\epsilon I) B-\lambda I|=|B(A-\epsilon I)-\lambda I|.\]
Taking the limit $\epsilon \to 0$, we obtain
\[|AB-\lambda I|=|BA-\lambda I|.\]
Comment.
When you study hard linear algebra, you might have forgotten about calculus.
This problem suggests that for some problems in linear algebra, techniques from calculus (like limits) might help to solve them.
Find the Limit of a Matrix
Let
\[A=\begin{bmatrix}
\frac{1}{7} & \frac{3}{7} & \frac{3}{7} \\
\frac{3}{7} &\frac{1}{7} &\frac{3}{7} \\
\frac{3}{7} & \frac{3}{7} & \frac{1}{7}
\end{bmatrix}\]
be $3 \times 3$ matrix. Find
\[\lim_{n \to \infty} A^n.\]
(Nagoya University Linear […]
Eigenvalues of Squared Matrix and Upper Triangular Matrix
Suppose that $A$ and $P$ are $3 \times 3$ matrices and $P$ is invertible matrix.
If
\[P^{-1}AP=\begin{bmatrix}
1 & 2 & 3 \\
0 &4 &5 \\
0 & 0 & 6
\end{bmatrix},\]
then find all the eigenvalues of the matrix $A^2$.
We give two proofs. The first version is a […]
Perturbation of a Singular Matrix is Nonsingular
Suppose that $A$ is an $n\times n$ singular matrix.
Prove that for sufficiently small $\epsilon>0$, the matrix $A-\epsilon I$ is nonsingular, where $I$ is the $n \times n$ identity matrix.
Hint.
Consider the characteristic polynomial $p(t)$ of the matrix $A$.
Note […]
Find All the Eigenvalues of Power of Matrix and Inverse Matrix
Let
\[A=\begin{bmatrix}
3 & -12 & 4 \\
-1 &0 &-2 \\
-1 & 5 & -1
\end{bmatrix}.\]
Then find all eigenvalues of $A^5$. If $A$ is invertible, then find all the eigenvalues of $A^{-1}$.
Proof.
We first determine all the eigenvalues of the matrix […]
Diagonalize the 3 by 3 Matrix if it is Diagonalizable
Determine whether the matrix
\[A=\begin{bmatrix}
0 & 1 & 0 \\
-1 &0 &0 \\
0 & 0 & 2
\end{bmatrix}\]
is diagonalizable.
If it is diagonalizable, then find the invertible matrix $S$ and a diagonal matrix $D$ such that $S^{-1}AS=D$.
How to […]
True of False Problems on Determinants and Invertible Matrices
Determine whether each of the following statements is True or False.
(a) If $A$ and $B$ are $n \times n$ matrices, and $P$ is an invertible $n \times n$ matrix such that $A=PBP^{-1}$, then $\det(A)=\det(B)$.
(b) If the characteristic polynomial of an $n \times n$ matrix $A$ […]
Nilpotent Matrices and Non-Singularity of Such Matrices
Let $A$ be an $n \times n$ nilpotent matrix, that is, $A^m=O$ for some positive integer $m$, where $O$ is the $n \times n$ zero matrix.
Prove that $A$ is a singular matrix and also prove that $I-A, I+A$ are both nonsingular matrices, where $I$ is the $n\times n$ identity […]