We first calculate the trace of the matrix $A$ as follows. We have
\begin{align*}
\tr(A)&=\tr(AB-BA)\\
&=\tr(AB)-\tr(BA)\\
&=\tr(AB)-\tr(AB)=0.
\end{align*}

Thus $\tr(A)=0$ and it follows from the Cayley-Hamilton theorem (see below) for the $2\times 2$ matrix $A$ that
\begin{align*}
O&=A^2-\tr(A)A+\det(A)I\\
&=A^2+\det(A)I,
\end{align*}
where $I$ is the $2\times 2$ identity matrix.

Thus, we obtain
\[A^2=-\det(A)I. \tag{*}\]

Next, we compute $A^2$ in two ways.
We have
\begin{align*}
A^2=A(AB-BA)=A^2B-ABA
\end{align*}
and
\begin{align*}
A^2=(AB-BA)A=ABA-BA^2.
\end{align*}
Adding these two, we have
\begin{align*}
2A^2&=A^2B-BA^2\\
& \stackrel{(*)}{=} (-\det(A)I)B-B(-\det(A)I)\\
&=-\det(A)B+\det(A)B=O.
\end{align*}

As a result, we obtain $A^2=O$. This completes the proof.

The Cayley-Hamilton theorem for a $2\times 2$ matrix

Let us add the proof of the fact we used in the proof about the Cayley-Hamilton theorem.
Let $A=\begin{bmatrix}
a & b\\
c& d
\end{bmatrix}$ be a $2\times 2$ matrix.

Then its characteristic polynomial is
\begin{align*}
p(x)&=\det(A-xI)\\
&=\begin{vmatrix}
a-x & b\\
c& d-x
\end{vmatrix}\\
&=(a-x)(d-x)-bc\\
&=x^2-(a+d)x+ad-bc\\
&=x^2-\tr(A)x+\det(A),
\end{align*}
since $\tr(A)=a+d$ and $\det(A)=ad-bc$.

The Cayley-Hamilton theorem says that the matrix $A$ satisfies its characteristic equation $p(x)=0$.
Namely we have
\[A^2-\tr(A)A+\det(A)I=O.\]
This is the equality we used in the proof.

Variation

As a variation of this problem, consider the following problem.

Let $A, B$ be $2\times 2$ matrices satisfying $A=AB-BA$.
Then prove that $\det(A)=0$.

True or False: If $A, B$ are 2 by 2 Matrices such that $(AB)^2=O$, then $(BA)^2=O$
Let $A$ and $B$ be $2\times 2$ matrices such that $(AB)^2=O$, where $O$ is the $2\times 2$ zero matrix.
Determine whether $(BA)^2$ must be $O$ as well. If so, prove it. If not, give a counter example.
Proof.
It is true that the matrix $(BA)^2$ must be the zero […]

If Two Matrices are Similar, then their Determinants are the Same
Prove that if $A$ and $B$ are similar matrices, then their determinants are the same.
Proof.
Suppose that $A$ and $B$ are similar. Then there exists a nonsingular matrix $S$ such that
\[S^{-1}AS=B\]
by definition.
Then we […]

Determine Whether Given Matrices are Similar
(a) Is the matrix $A=\begin{bmatrix}
1 & 2\\
0& 3
\end{bmatrix}$ similar to the matrix $B=\begin{bmatrix}
3 & 0\\
1& 2
\end{bmatrix}$?
(b) Is the matrix $A=\begin{bmatrix}
0 & 1\\
5& 3
\end{bmatrix}$ similar to the matrix […]

Trace, Determinant, and Eigenvalue (Harvard University Exam Problem)
(a) A $2 \times 2$ matrix $A$ satisfies $\tr(A^2)=5$ and $\tr(A)=3$.
Find $\det(A)$.
(b) A $2 \times 2$ matrix has two parallel columns and $\tr(A)=5$. Find $\tr(A^2)$.
(c) A $2\times 2$ matrix $A$ has $\det(A)=5$ and positive integer eigenvalues. What is the trace of […]

The Formula for the Inverse Matrix of $I+A$ for a $2\times 2$ Singular Matrix $A$
Let $A$ be a singular $2\times 2$ matrix such that $\tr(A)\neq -1$ and let $I$ be the $2\times 2$ identity matrix.
Then prove that the inverse matrix of the matrix $I+A$ is given by the following formula:
\[(I+A)^{-1}=I-\frac{1}{1+\tr(A)}A.\]
Using the formula, calculate […]

An Example of a Matrix that Cannot Be a Commutator
Let $I$ be the $2\times 2$ identity matrix.
Then prove that $-I$ cannot be a commutator $[A, B]:=ABA^{-1}B^{-1}$ for any $2\times 2$ matrices $A$ and $B$ with determinant $1$.
Proof.
Assume that $[A, B]=-I$. Then $ABA^{-1}B^{-1}=-I$ implies
\[ABA^{-1}=-B. […]

How to Use the Cayley-Hamilton Theorem to Find the Inverse Matrix
Find the inverse matrix of the $3\times 3$ matrix
\[A=\begin{bmatrix}
7 & 2 & -2 \\
-6 &-1 &2 \\
6 & 2 & -1
\end{bmatrix}\]
using the Cayley-Hamilton theorem.
Solution.
To apply the Cayley-Hamilton theorem, we first determine the characteristic […]

Matrix $XY-YX$ Never Be the Identity Matrix
Let $I$ be the $n\times n$ identity matrix, where $n$ is a positive integer. Prove that there are no $n\times n$ matrices $X$ and $Y$ such that
\[XY-YX=I.\]
Hint.
Suppose that such matrices exist and consider the trace of the matrix $XY-YX$.
Recall that the trace of […]