Since $A$ is a real matrix and $\frac{-1+\sqrt{3}i}{2}$ is a complex eigenvalue, its conjugate $\frac{-1-\sqrt{3}i}{2}$ is also an eigenvalue of $A$.
As $A$ is a $3\times 3$ matrix, it has one more eigenvalue $\lambda$.

Note that the product of all eigenvalues of $A$ is the determinant of $A$.
Thus, we have
\[\frac{-1+\sqrt{3}i}{2} \cdot \frac{-1-\sqrt{3}i}{2}\cdot \lambda =\det(A)=1.\]
Solving this, we obtain $\lambda=1$.
Therefore, the eigenvalues of $A$ are
\[\frac{-1+\sqrt{3}i}{2}, \frac{-1-\sqrt{3}i}{2}, 1.\]

(a) Using the Cayley-Hamilton theorem, determine $a, b, c$.

To use the Cayley-Hamilton theorem, we first need to determine the characteristic polynomial $p(t)=\det(A-tI)$ of $A$.
Since we found all the eigenvalues of $A$ in part (a) and the roots of characteristic polynomials are the eigenvalues, we know that
\begin{align*}
p(t)&=-\left(\, t-\frac{-1+\sqrt{3}i}{2} \,\right)\left(\, t-\frac{-1-\sqrt{3}i}{2} \,\right)(t-1) \tag{*}\\
&=-(t^2+t+1)(t-1)\\
&=-t^3+1.
\end{align*}
(Remark that if your definition of the characteristic polynomial is $\det(tI-A)$, then the first negative sign in (*) should be omitted.)

Then the Cayley-Hamilton theorem yields that
\[P(A)=-A^3+I=O,\]
where $O$ is the $3\times 3$ zero matrix.

Hence we have $A^3=I$.
We compute
\begin{align*}
A^{100}=(A^3)^{33}A=I^{33}A=IA=A.
\end{align*}

Thus, we conclude that $a=0, b=1, c=0$.

Comment.

Observe that we did not use the assumption that $A$ is orthogonal.

Find the Eigenvalues and Eigenvectors of the Matrix $A^4-3A^3+3A^2-2A+8E$.
Let
\[A=\begin{bmatrix}
1 & -1\\
2& 3
\end{bmatrix}.\]
Find the eigenvalues and the eigenvectors of the matrix
\[B=A^4-3A^3+3A^2-2A+8E.\]
(Nagoya University Linear Algebra Exam Problem)
Hint.
Apply the Cayley-Hamilton theorem.
That is if $p_A(t)$ is the […]

A Square Root Matrix of a Symmetric Matrix
Answer the following two questions with justification.
(a) Does there exist a $2 \times 2$ matrix $A$ with $A^3=O$ but $A^2 \neq O$? Here $O$ denotes the $2 \times 2$ zero matrix.
(b) Does there exist a $3 \times 3$ real matrix $B$ such that $B^2=A$ […]

Find All the Eigenvalues of 4 by 4 Matrix
Find all the eigenvalues of the matrix
\[A=\begin{bmatrix}
0 & 1 & 0 & 0 \\
0 &0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0
\end{bmatrix}.\]
(The Ohio State University, Linear Algebra Final Exam Problem)
Solution.
We compute the […]

Diagonalize a 2 by 2 Matrix $A$ and Calculate the Power $A^{100}$
Let
\[A=\begin{bmatrix}
1 & 2\\
4& 3
\end{bmatrix}.\]
(a) Find eigenvalues of the matrix $A$.
(b) Find eigenvectors for each eigenvalue of $A$.
(c) Diagonalize the matrix $A$. That is, find an invertible matrix $S$ and a diagonal matrix $D$ such that […]

Diagonalize the 3 by 3 Matrix if it is Diagonalizable
Determine whether the matrix
\[A=\begin{bmatrix}
0 & 1 & 0 \\
-1 &0 &0 \\
0 & 0 & 2
\end{bmatrix}\]
is diagonalizable.
If it is diagonalizable, then find the invertible matrix $S$ and a diagonal matrix $D$ such that $S^{-1}AS=D$.
How to […]

Rotation Matrix in Space and its Determinant and Eigenvalues
For a real number $0\leq \theta \leq \pi$, we define the real $3\times 3$ matrix $A$ by
\[A=\begin{bmatrix}
\cos\theta & -\sin\theta & 0 \\
\sin\theta &\cos\theta &0 \\
0 & 0 & 1
\end{bmatrix}.\]
(a) Find the determinant of the matrix $A$.
(b) Show that $A$ is an […]

How to Calculate and Simplify a Matrix Polynomial
Let $T=\begin{bmatrix}
1 & 0 & 2 \\
0 &1 &1 \\
0 & 0 & 2
\end{bmatrix}$.
Calculate and simplify the expression
\[-T^3+4T^2+5T-2I,\]
where $I$ is the $3\times 3$ identity matrix.
(The Ohio State University exam)
Hint.
Use the […]