A WARNING. Almost everything that I know about using barycentric coordinates and using matrices to solve linear equations was learned last night because I found this question so interesting. So the following may be wrong, wrong, wrong - but some of the test values that I set do work. Guys and girls, please feel free to tear it apart if I completely messed up, but here it goes.
Search for barycentric coords in 3D space (with a little help from Wikipedia)
Given:
v0 = (x0, y0, z0) v1 = (x1, y1, z1) v2 = (x2, y2, z2) p = (xp, yp, zp)
Find the barycentric coordinates: b0, b1, b2 of the point p relative to the triangle defined by v0, v1 and v2
Knowing that:
xp = b0*x0 + b1*x1 + b2*x2 yp = b0*y0 + b1*y1 + b2*y2 zp = b0*z0 + b1*z1 + b2*z2
What can be written as
[xp] [x0] [x1] [x2] [yp] = b0*[y0] + b1*[y1] + b2*[y2] [zp] [z0] [z1] [z2]
or
[xp] [x0 x1 x2] [b0] [yp] = [y0 y1 y2] . [b1] [zp] [z0 z1 z2] [b2]
rebuilt as
-1 [b0] [x0 x1 x2] [xp] [b1] = [y0 y1 y2] . [yp] [b2] [z0 z1 z2] [zp]
3x3 matrix determinant:
det = x0(y1*z2 - y2*z1) + x1(y2*z0 - z2*y0) + x2(y0*z1 - y1*z0)
its conjugate is
[y1*z2-y2*z1 x2*z1-x1*z2 x1*y2-x2*y1] [y2*z0-y0*z2 x0*z2-x2*z0 x2*y0-x0*y2] [y0*z1-y1*z0 x1*z0-x0*z1 x0*y1-x1*y0]
giving:
[b0] [y1*z2-y2*z1 x2*z1-x1*z2 x1*y2-x2*y1] [xp] [b1] = ( [y2*z0-y0*z2 x0*z2-x2*z0 x2*y0-x0*y2] . [yp] ) / det [b2] [y0*z1-y1*z0 x1*z0-x0*z1 x0*y1-x1*y0] [zp]
If you need to check several points against a triangle, stop here. Calculate the above 3x3 matrix once for a triangle (dividing it also by the determinant), and then the point product, which will cause each point to receive barycentric coordinates for each point.
If you do this only once per triangle, then the above is multiplied (kindly provided by Maxima):
b0 = ((x1*y2-x2*y1)*zp+xp*(y1*z2-y2*z1)+yp*(x2*z1-x1*z2)) / det b1 = ((x2*y0-x0*y2)*zp+xp*(y2*z0-y0*z2)+yp*(x0*z2-x2*z0)) / det b2 = ((x0*y1-x1*y0)*zp+xp*(y0*z1-y1*z0)+yp*(x1*z0-x0*z1)) / det
These are quite a few additions, subtractions and multiplications - three divisions, but not sqrts or trigger functions. Obviously, this takes longer than pure 2D calculations, but depending on the complexity of the projection heuristic and calculations, this may turn out to be the fastest way.
As I mentioned - I have no idea what I'm talking about, but maybe it will work, or maybe someone else can come and fix it.