It seems you need some kind of parameterization of the SO(3) group in Mathematica, I think. You will have only 3 independent symbols (variables), since you have 6 restrictions on the mutual orthogonality of vectors and norms equal to 1. One way is to build independent rotations around three axes and multiply these matrices. Here's the (possibly too complicated) code for this:
makeOrthogonalMatrix[p_Symbol, q_Symbol, t_Symbol] := Module[{permute, matrixGeneratingFunctions}, permute = Function[perm, Permute[Transpose[Permute[#, perm]], perm] &]; matrixGeneratingFunctions = Function /@ FoldList[ permute[#2][#1] &, {{Cos[#], 0, Sin[#]}, {0, 1, 0}, {-Sin[#], 0, Cos[#]}}, {{2, 1, 3}, {3, 2, 1}}]; #1.#2.#3 & @@ MapThread[Compose, {matrixGeneratingFunctions, {p, q, t}}]];
Here's how it works:
In[62]:= makeOrthogonalMatrix[x,y,z] Out[62]= {{Cos[x] Cos[z]+Sin[x] Sin[y] Sin[z],Cos[z] Sin[x] Sin[y]-Cos[x] Sin[z],Cos[y] Sin[x]}, {Cos[y] Sin[z],Cos[y] Cos[z],-Sin[y]}, {-Cos[z] Sin[x]+Cos[x] Sin[y] Sin[z],Cos[x] Cos[z] Sin[y]+Sin[x] Sin[z],Cos[x] Cos[y]}}
You can verify that the matrix is โโorthonormalized using Simplify for various point products of columns (or rows).