Assume that $A$ is nonsingular. Then the inverse matrix $A^{-1}$ exists.
Multiplying the given equality by $A^{-1}$ on the left, we obtain
\[3A^{-1}A=A^{-1}(A^2+AB)=A^{-1}A^2+A^{-1}AB=A+IB=A+B.\]
Note that the left most term is equal to $3I$.
Hence, we have $3I=A+B$. Solving for $A$, we have
\[A=3I-B=\begin{bmatrix}
3 & 0 & 0 \\
0 &3 &0 \\
0 & 0 & 3
\end{bmatrix}-\begin{bmatrix}
2 & 0 & -1 \\
0 &2 &-1 \\
-1 & 0 & 1
\end{bmatrix}=\begin{bmatrix}
1 & 0 & 1 \\
0 &1 &1 \\
1 & 0 & 2
\end{bmatrix}.\]

As we see in part (b), this matrix is actually invertible.

(b) Find the inverse matrix of $A$.

To find the inverse matrix of $A$, we reduce the augmented matrix $[A\mid I]$ as follows:
\begin{align*}
[A\mid I]= \left[\begin{array}{rrr|rrr}
1 & 0 & 1 & 1 &0 & 0 \\
0 & 1 & 1 & 0 & 1 & 0 \\
1 & 0 & 2 & 0 & 0 & 1 \\
\end{array} \right]
\xrightarrow{R_3-R_2}
\left[\begin{array}{rrr|rrr}
1 & 0 & 1 & 1 &0 & 0 \\
0 & 1 & 1 & 0 & 1 & 0 \\
0 & 0 & 1 & -1 & 0 & 1 \\
\end{array} \right]\\[6pt]
\xrightarrow[R_2-R_3]{R_1-R_3}
\left[\begin{array}{rrr|rrr}
1 & 0 & 0 & 2 &0 & -1 \\
0 & 1 & 0 & 1 & 1 & -1 \\
0 & 0 & 1 & -1 & 0 & 1 \\
\end{array} \right].
\end{align*}
The left 3 by 3 part is now the identity matrix.
So the inverse matrix is given by the right 3 by 3 part:
\[A^{-1}=\begin{bmatrix}
2 & 0 & -1 \\
1 &1 &-1 \\
-1 & 0 & 1
\end{bmatrix}.\]

Common Mistake

This is a midterm exam problem of Lienar Algebra at the Ohio State University.

One common mistake is: $A=3-B$. Don’t forget the identity matrix $I$. $3$ is a number but $3I$ is a matrix.
Also $3I$ is the 3 by 3 matrix whose diagonal entries are 3 and 0 elsewhere. Note that $3I$ is not the 3 by3 matrix whose entries are all 3.

Quiz 4: Inverse Matrix/ Nonsingular Matrix Satisfying a Relation
(a) Find the inverse matrix of
\[A=\begin{bmatrix}
1 & 0 & 1 \\
1 &0 &0 \\
2 & 1 & 1
\end{bmatrix}\]
if it exists. If you think there is no inverse matrix of $A$, then give a reason.
(b) Find a nonsingular $2\times 2$ matrix $A$ such that
\[A^3=A^2B-3A^2,\]
where […]

Find the Inverse Matrix of a $3\times 3$ Matrix if Exists
Find the inverse matrix of
\[A=\begin{bmatrix}
1 & 1 & 2 \\
0 &0 &1 \\
1 & 0 & 1
\end{bmatrix}\]
if it exists. If you think there is no inverse matrix of $A$, then give a reason.
(The Ohio State University, Linear Algebra Midterm Exam […]

Using Properties of Inverse Matrices, Simplify the Expression
Let $A, B, C$ be $n\times n$ invertible matrices. When you simplify the expression
\[C^{-1}(AB^{-1})^{-1}(CA^{-1})^{-1}C^2,\]
which matrix do you get?
(a) $A$
(b) $C^{-1}A^{-1}BC^{-1}AC^2$
(c) $B$
(d) $C^2$
(e) $C^{-1}BC$
(f) $C$
Solution.
In this problem, we […]

True or False Problems on Midterm Exam 1 at OSU Spring 2018
The following problems are True or False.
Let $A$ and $B$ be $n\times n$ matrices.
(a) If $AB=B$, then $B$ is the identity matrix.
(b) If the coefficient matrix $A$ of the system $A\mathbf{x}=\mathbf{b}$ is invertible, then the system has infinitely many solutions.
(c) If $A$ […]

Diagonalize a 2 by 2 Matrix if Diagonalizable
Determine whether the matrix
\[A=\begin{bmatrix}
1 & 4\\
2 & 3
\end{bmatrix}\]
is diagonalizable.
If so, find a nonsingular matrix $S$ and a diagonal matrix $D$ such that $S^{-1}AS=D$.
(The Ohio State University, Linear Algebra Final Exam […]

Find a Nonsingular Matrix Satisfying Some Relation
Determine whether there exists a nonsingular matrix $A$ if
\[A^2=AB+2A,\]
where $B$ is the following matrix.
If such a nonsingular matrix $A$ exists, find the inverse matrix $A^{-1}$.
(a) \[B=\begin{bmatrix}
-1 & 1 & -1 \\
0 &-1 &0 \\
1 & 2 & […]

Solve a System by the Inverse Matrix and Compute $A^{2017}\mathbf{x}$
Let $A$ be the coefficient matrix of the system of linear equations
\begin{align*}
-x_1-2x_2&=1\\
2x_1+3x_2&=-1.
\end{align*}
(a) Solve the system by finding the inverse matrix $A^{-1}$.
(b) Let $\mathbf{x}=\begin{bmatrix}
x_1 \\
x_2
\end{bmatrix}$ be the solution […]

Determine Whether There Exists a Nonsingular Matrix Satisfying $A^4=ABA^2+2A^3$
Determine whether there exists a nonsingular matrix $A$ if
\[A^4=ABA^2+2A^3,\]
where $B$ is the following matrix.
\[B=\begin{bmatrix}
-1 & 1 & -1 \\
0 &-1 &0 \\
2 & 1 & -4
\end{bmatrix}.\]
If such a nonsingular matrix $A$ exists, find the inverse […]