It is true that the matrix $(BA)^2$ must be the zero matrix as we shall prove now.

For notational convenience, let $C:=AB$.

As $C$ is a $2\times 2$ matrix, it satisfies the relation
\[C^2-\tr(C)C+\det(C)I=O \tag{*}\]
by the Cayley-Hamilton theorem.
Here $I$ is the $2\times 2$ identity matrix.

We compute the determinant of $C$ as follows.
We have
\begin{align*}
\det(C)^2=\det(C^2)=\det((AB)^2)=\det(O)=0.
\end{align*}
Hence $\det(C)=0$.
Since $C^2=O$ and $\det(C)=0$, the Cayley-Hamilton relation (*) becomes
\[\tr(C)C=O.\]
It follows that we have either $C=O$ or $\tr(C)=0$.
In either case, we have $\tr(C)=0$.
Note that
\begin{align*}
\det(BA)&=\det(AB)=\det(C)=0\\
\tr(BA)&=\tr(AB)=\tr(C)=0.
\end{align*}

Applying the Cayley-Hamilton theorem to the matrix $BA$, we obtain
\begin{align*}
O=(BA)^2-\tr(BA)\cdot BA+\det(BA)I=(BA)^2.
\end{align*}
Thus, we obtain $(BA)^2=O$ as claimed.

If 2 by 2 Matrices Satisfy $A=AB-BA$, then $A^2$ is Zero Matrix
Let $A, B$ be complex $2\times 2$ matrices satisfying the relation
\[A=AB-BA.\]
Prove that $A^2=O$, where $O$ is the $2\times 2$ zero matrix.
Hint.
Find the trace of $A$.
Use the Cayley-Hamilton theorem
Proof.
We first calculate the […]

If Two Matrices are Similar, then their Determinants are the Same
Prove that if $A$ and $B$ are similar matrices, then their determinants are the same.
Proof.
Suppose that $A$ and $B$ are similar. Then there exists a nonsingular matrix $S$ such that
\[S^{-1}AS=B\]
by definition.
Then we […]

Determine Whether Given Matrices are Similar
(a) Is the matrix $A=\begin{bmatrix}
1 & 2\\
0& 3
\end{bmatrix}$ similar to the matrix $B=\begin{bmatrix}
3 & 0\\
1& 2
\end{bmatrix}$?
(b) Is the matrix $A=\begin{bmatrix}
0 & 1\\
5& 3
\end{bmatrix}$ similar to the matrix […]

Trace, Determinant, and Eigenvalue (Harvard University Exam Problem)
(a) A $2 \times 2$ matrix $A$ satisfies $\tr(A^2)=5$ and $\tr(A)=3$.
Find $\det(A)$.
(b) A $2 \times 2$ matrix has two parallel columns and $\tr(A)=5$. Find $\tr(A^2)$.
(c) A $2\times 2$ matrix $A$ has $\det(A)=5$ and positive integer eigenvalues. What is the trace of […]

The Formula for the Inverse Matrix of $I+A$ for a $2\times 2$ Singular Matrix $A$
Let $A$ be a singular $2\times 2$ matrix such that $\tr(A)\neq -1$ and let $I$ be the $2\times 2$ identity matrix.
Then prove that the inverse matrix of the matrix $I+A$ is given by the following formula:
\[(I+A)^{-1}=I-\frac{1}{1+\tr(A)}A.\]
Using the formula, calculate […]

An Example of a Matrix that Cannot Be a Commutator
Let $I$ be the $2\times 2$ identity matrix.
Then prove that $-I$ cannot be a commutator $[A, B]:=ABA^{-1}B^{-1}$ for any $2\times 2$ matrices $A$ and $B$ with determinant $1$.
Proof.
Assume that $[A, B]=-I$. Then $ABA^{-1}B^{-1}=-I$ implies
\[ABA^{-1}=-B. […]

Matrix $XY-YX$ Never Be the Identity Matrix
Let $I$ be the $n\times n$ identity matrix, where $n$ is a positive integer. Prove that there are no $n\times n$ matrices $X$ and $Y$ such that
\[XY-YX=I.\]
Hint.
Suppose that such matrices exist and consider the trace of the matrix $XY-YX$.
Recall that the trace of […]

This problem can also be done in this way. Let C=AB and D=BA. Then both C,D are 2 by 2 matrices and according to given condition C^2=0. Hence C is a nilpotent matrix. So all eigenvalues of C=AB are 0 and we know for square matrices,AB and BA have same set of eigenvalues, So all eigenvalues of BA are zero. Suppose p(t) be the characteristic polynomial of D=BA. Then we have p(t)=t^2. And by cayley hamilton theorem, we have p(D)=0 or D^2=0 or (BA)^2=0.

This problem can also be done in this way. Let C=AB and D=BA. Then both C,D are 2 by 2 matrices and according to given condition C^2=0. Hence C is a nilpotent matrix. So all eigenvalues of C=AB are 0 and we know for square matrices,AB and BA have same set of eigenvalues, So all eigenvalues of BA are zero. Suppose p(t) be the characteristic polynomial of D=BA. Then we have p(t)=t^2. And by cayley hamilton theorem, we have p(D)=0 or D^2=0 or (BA)^2=0.