I have not used GLM, so you may not like this answer. However, performing a quaternion rotation is not bad.
Say your camera has the initial saved orientation of 'vecOriginalDirection' (normalized vec3). Let's say you want it to follow another "vecDirection" (also normalized). Thus, we can adapt an approach similar to a trackball and consider vecDirection as a deviation from what is the default focus of the camera.
Usually the preferred way to rotate a quaternion in the real world is to use NLERP. Let's see if I remember: in the pseudocode (assuming floating point), I think this is:
quat = normalize([ cross(vecDirection, vecOriginalDirection), 1. + dot(vecDirection, vecOriginalDirection)]);
(Donβt forget that β1. +β, I forgot why this happened, but it made sense at the same time. I think I pulled my hair out for a few days until I found it. It's basically a block quaternion, IIRC, which is averaged, thereby making the double-angle action similar to a corner ... maybe :))
The renormalization shown above as "normalize ()" is significant (this is "N" in NLERP). Of course, the normalization of quat (x, y, z, w) is true:
quat /= sqrt(x*x+y*y+z*z+w*w);
Then, if you want to use your own function to create a 3x3 orientation matrix from quat:
xx=2.*x*x, yy=2.*y*y, zz=2.*z*z, xy=2.*x*y, xz=2.*x*z, yz=2.*y*z, wx=2.*w*x, wy=2.*w*y, wz=2.*w*z; m[0]=1.-(yy+zz); m[1]=xy+wz; m[2]=xz-wy; m[3]=xy-wz; m[4]=1.-(xx+zz); m[5]=yz+wx; m[6]=xz+wy; m[7]=yz-wx; m[8]=1.-(xx+yy);
To actually implement the trackball, you will need to calculate vecDirection when the finger is held, and save it on vecOriginalDirection when it is first pressed (subject to the touch interface).
You will also probably want to calculate these values ββbased on the piecewise function of the hemisphere / hyperboloid, if you have not already done so. I think that @minorlogic was trying to save some time, since it looks like you could just use a virtual trackball with a drop down list.