I do not use Eigen and did not bother looking for an API, but here is a simple, inexpensive and stable procedure for re-orthogonalizing a rotation matrix. This orthogonalization procedure is taken from the IMU Cosine Matrix Direction: Theory
William Premerlani and Paul Bager; equations 19-21.
Let x, yand zbe the row vectors of the (slightly spoiled) rotation matrix. Let error=dot(x,y)where dot()- a point product. If the matrix is orthogonal, the point product xand y, i.e. errorwill be zero.
error x y: x_ort=x-(error/2)*y y_ort=y-(error/2)*x. z_ort=cross(x_ort, y_ort), x_ort y_ort.
x_ort, y_ort z_ort, .
x_new = 1/2*(3-dot(x_ort,x_ort))*x_ort
y_new = 1/2*(3-dot(y_ort,y_ort))*y_ort
z_new = 1/2*(3-dot(z_ort,z_ort))*z_ort
.
API, Eigen. , , . , ; .