Since $A$ is singular, the equation $A\mathbf{x}=\mathbf{0}$ has a nonzero solution.
Let $\mathbf{x}_1$ be a nonzero solution of $A\mathbf{x}=\mathbf{0}$.
We define the $3\times 3$ matrix $B$ to be
\[B=[\mathbf{x}_1, \mathbf{0}, \mathbf{0}],\]
that is, the first column of $B$ is the vector $x_1$ and the second and the third column vectors are $3$-dimensional zero vectors.

Then since $x_1\neq \mathbf{0}$, the matrix $B$ is not the zero matrix.
We have
\begin{align*}
AB&=A[\mathbf{x}_1, \mathbf{0}, \mathbf{0}]\\
&=[A\mathbf{x}_1, A\mathbf{0}, A\mathbf{0}]\\
&=[\mathbf{0}, \mathbf{0}, \mathbf{0}]=O,
\end{align*}
since $A\mathbf{x}_1=\mathbf{0}$.

Thus we have found the nonzero matrix $B$ such that the product $AB=O$.

Comment.

This is one of the midterm exam 1 problems of linear algebra (Math 2568) at the Ohio State University.

In the exam the following hint was given:

Hint: Let $B=[\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3]$, where $\mathbf{x}_i$ is the $i$-th column vector of $B$ for $i=1,2,3$. Then $AB=[A\mathbf{x}_1, A\mathbf{x}_2, A\mathbf{x}_3]$.

Common Mistake

Several students wrote “Since $A$ is singular, $A$ has a solution”.
This does not make sense. A solution of what? You need to remember the definition correctly.

A matrix $A$ is singular if the equation $A\mathbf{x}=\mathbf{0}$ has a nonzero solution $\mathbf{x}$.

Also, note that $\mathbf{x}$ is a vector, not a number.

Midterm 1 problems and solutions

The following list is the problems and solutions/proofs of midterm exam 1 of linear algebra at the Ohio State University in Spring 2017.

10 True or False Problems about Basic Matrix Operations
Test your understanding of basic properties of matrix operations.
There are 10 True or False Quiz Problems.
These 10 problems are very common and essential.
So make sure to understand these and don't lose a point if any of these is your exam problems.
(These are actual exam […]

Eigenvalues of a Hermitian Matrix are Real Numbers
Show that eigenvalues of a Hermitian matrix $A$ are real numbers.
(The Ohio State University Linear Algebra Exam Problem)
We give two proofs. These two proofs are essentially the same.
The second proof is a bit simpler and concise compared to the first one.
[…]

Given All Eigenvalues and Eigenspaces, Compute a Matrix Product
Let $C$ be a $4 \times 4$ matrix with all eigenvalues $\lambda=2, -1$ and eigensapces
\[E_2=\Span\left \{\quad \begin{bmatrix}
1 \\
1 \\
1 \\
1
\end{bmatrix} \quad\right \} \text{ and } E_{-1}=\Span\left \{ \quad\begin{bmatrix}
1 \\
2 \\
1 \\
1
[…]

A Matrix Representation of a Linear Transformation and Related Subspaces
Let $T:\R^4 \to \R^3$ be a linear transformation defined by
\[ T\left (\, \begin{bmatrix}
x_1 \\
x_2 \\
x_3 \\
x_4
\end{bmatrix} \,\right) = \begin{bmatrix}
x_1+2x_2+3x_3-x_4 \\
3x_1+5x_2+8x_3-2x_4 \\
x_1+x_2+2x_3
\end{bmatrix}.\]
(a) Find a matrix $A$ such that […]

Compute the Product $A^{2017}\mathbf{u}$ of a Matrix Power and a Vector
Let
\[A=\begin{bmatrix}
-1 & 2 \\
0 & -1
\end{bmatrix} \text{ and } \mathbf{u}=\begin{bmatrix}
1\\
0
\end{bmatrix}.\]
Compute $A^{2017}\mathbf{u}$.
(The Ohio State University, Linear Algebra Exam)
Solution.
We first compute $A\mathbf{u}$. We […]

Find the Nullity of the Matrix $A+I$ if Eigenvalues are $1, 2, 3, 4, 5$
Let $A$ be an $n\times n$ matrix. Its only eigenvalues are $1, 2, 3, 4, 5$, possibly with multiplicities.
What is the nullity of the matrix $A+I_n$, where $I_n$ is the $n\times n$ identity matrix?
(The Ohio State University, Linear Algebra Final Exam […]

Linear Transformation and a Basis of the Vector Space $\R^3$
Let $T$ be a linear transformation from the vector space $\R^3$ to $\R^3$.
Suppose that $k=3$ is the smallest positive integer such that $T^k=\mathbf{0}$ (the zero linear transformation) and suppose that we have $\mathbf{x}\in \R^3$ such that $T^2\mathbf{x}\neq \mathbf{0}$.
Show […]

There is an error. The problem says A is singular. The proof says A is nonsingular.

Dear Michael,

Thank you for pointing out the typo. I fixed it.