Firstly, you do not want to use gluLookAt . gluLookAt rotates the camera, but the physical screen the user is looking at does not rotate. gluLookAt will only work if the screen gluLookAt so that the normal screen continues to point to the user. The perspective distortion of the off-axis projection will take care of all the turns we need.
What you need to consider in your model is the position of the screen inside the truncated cone. Consider the following image. Red dots are the borders of the screen. What you need to achieve is that these positions remain constant in 3D WCS, since the physical screen in the real world also (hopefully) does not move. I think this is a key understanding of virtual reality and stereoscopy. A screen is something like a window into virtual reality and alignment of the real world with virtual reality, you need to align the truncated angle with this window.
To do this, you need to determine the position of the screen in the Kinect coordinate system. Assuming Kinect is on top of the screen, that + y is pointing down, and that the unit you are using is millimeters, I would expect these coordinates to be something like lines (+ -300, 200, 0), (+ -300, 500, 0).
Now for a distant plane there are two possibilities. You can either choose a fixed distance from the camera to the far. This would mean that the distant plane would move back if the user moved back, possibly cropping the objects you want to draw. Or you can hold the far plane in a fixed position in the WCS, as shown in the image. I find the latter more useful. For the near plane, I think the fixed distance from the camera is fine.
The inputs are the three-dimensional screen positions wcsPtTopLeftScreen and wcsPtBottomRightScreen , the tracked head position wcsPtHead , the z value of the far plane wcsZFar (all in WCS), and z the value of the near plane camZNear (in camera coordinates). We need to calculate the trimming parameters in the coordinates of the camera.
camPtTopLeftScreen = wcsPtTopLeftScreen - wcsPtHead; camPtTopLeftNear = camPtTopLeftScreen / camPtTopLeftScreen.z * camZNear;
and the same with the bottom right point. Also:
camZFar = wcsZFar - wcsPtHead.z
Now the only problem is that Kinect and OpenGL use different coordinate systems. In Kinect, CS + y points down, + z points from the user to Kinect. In OpenGL, + y points up, + z points to the viewer. This means that we must multiply y and z by -1:
glFrustum(camPtTopLeftNear.x, camPtBottomRightNear.x, -camPtBottomRightNear.y, -camPtTopLeftNear.y, camZNear, camZFar);
If you need a better explanation that also covers stereoscopy, watch this video , I found it insightful and well executed.
A quick demo, you may need to configure wcsWidth , pxWidth and wcsPtHead.z .
#include <glm/glm.hpp> #include <glm/ext.hpp> #include <glut.h> #include <functional> float heightFromWidth; glm::vec3 camPtTopLeftNear, camPtBottomRightNear; float camZNear, camZFar; glm::vec3 wcsPtHead(0, 0, -700); void moveCameraXY(int pxPosX, int pxPosY) { // Width of the screen in mm and in pixels. float wcsWidth = 520.0; float pxWidth = 1920.0f; float wcsHeight = heightFromWidth * wcsWidth; float pxHeight = heightFromWidth * pxWidth; float wcsFromPx = wcsWidth / pxWidth; glm::vec3 wcsPtTopLeftScreen(-wcsWidth/2.f, -wcsHeight/2.f, 0); glm::vec3 wcsPtBottomRightScreen(wcsWidth/2.f, wcsHeight/2.f, 0); wcsPtHead = glm::vec3(wcsFromPx * float(pxPosX - pxWidth / 2), wcsFromPx * float(pxPosY - pxHeight * 0.5f), wcsPtHead.z); camZNear = 1.0; float wcsZFar = 500; glm::vec3 camPtTopLeftScreen = wcsPtTopLeftScreen - wcsPtHead; camPtTopLeftNear = camZNear / camPtTopLeftScreen.z * camPtTopLeftScreen; glm::vec3 camPtBottomRightScreen = wcsPtBottomRightScreen - wcsPtHead; camPtBottomRightNear = camPtBottomRightScreen / camPtBottomRightScreen.z * camZNear; camZFar = wcsZFar - wcsPtHead.z; glutPostRedisplay(); } void moveCameraZ(int button, int state, int x, int y) { // No mouse wheel in GLUT. :( if ((button == 0) || (button == 2)) { if (state == GLUT_DOWN) return; wcsPtHead.z += (button == 0 ? -1 : 1) * 100; glutPostRedisplay(); } } void reshape(int w, int h) { heightFromWidth = float(h) / float(w); glViewport(0, 0, w, h); } void drawObject(std::function<void(GLdouble)> drawSolid, std::function<void(GLdouble)> drawWireframe, GLdouble size) { glPushAttrib(GL_ALL_ATTRIB_BITS); glEnable(GL_COLOR); glDisable(GL_LIGHTING); glColor4f(1, 1, 1, 1); drawSolid(size); glColor4f(0.8, 0.8, 0.8, 1); glDisable(GL_DEPTH_TEST); glLineWidth(1); drawWireframe(size); glColor4f(0, 0, 0, 1); glEnable(GL_DEPTH_TEST); glLineWidth(3); drawWireframe(size); glPopAttrib(); } void display(void) { glPushAttrib(GL_ALL_ATTRIB_BITS); glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT); glEnable(GL_DEPTH_TEST); // In the Kinect CS, +y points down, +z points from the user towards the Kinect. // In OpenGL, +y points up, +z points towards the viewer. glm::mat4 mvpCube; mvpCube = glm::frustum(camPtTopLeftNear.x, camPtBottomRightNear.x, -camPtBottomRightNear.y, -camPtTopLeftNear.y, camZNear, camZFar); mvpCube = glm::scale(mvpCube, glm::vec3(1, -1, -1)); mvpCube = glm::translate(mvpCube, -wcsPtHead); glMatrixMode(GL_MODELVIEW); glLoadMatrixf(glm::value_ptr(mvpCube)); drawObject(glutSolidCube, glutWireCube, 140); glm::mat4 mvpTeapot = glm::translate(mvpCube, glm::vec3(100, 0, 200)); mvpTeapot = glm::scale(mvpTeapot, glm::vec3(1, -1, -1)); // teapots are in OpenGL coordinates glLoadMatrixf(glm::value_ptr(mvpTeapot)); glColor4f(1, 1, 1, 1); drawObject(glutSolidTeapot, glutWireTeapot, 50); glFlush(); glPopAttrib(); } void leave(unsigned char, int, int) { exit(0); } int main(int argc, char **argv) { glutInit(&argc, argv); glutCreateWindow("glut test"); glutDisplayFunc(display); glutReshapeFunc(reshape); moveCameraXY(0,0); glutPassiveMotionFunc(moveCameraXY); glutMouseFunc(moveCameraZ); glutKeyboardFunc(leave); glutFullScreen(); glutMainLoop(); return 0; }
The following images should be viewed from a distance equal to 135% of their width on the screen (70 cm on my screen 52 cm wide in full screen).