Barycentric coordinates! More details:
Let the envelope triangle have vertices A, B, C; without loss of generality, you can place vertex A at the origin and align side AB with the + x axis. Use the lengths of the edges of the envelope triangle to find the angle at vertex A, that is, the angle between sides AB and AC. Using this angle, you define a new coordinate system (u, v); in this coordinate system, the vertex coordinates A = (0,0), B = (1,0) and C = (0,1).
Now take another triangle with vertices A ', B', C 'and first find the XY coordinates of the three vertices for each case: (A'B', B'C ', A'C') aligned with the + x coordinate axis. For each such alignment, transform the other two vertices into a UV coordinate system defined by the envelope triangle. If it happens that both other vertices have (u, v) coordinates with 0 <= u, v <= 1 with u + v <= 1, the triangle is placed in the envelope triangle.
The angle between the two sides can be obtained through the sine theorem for plane triangles; although you should be a little careful if the angle at the vertex is obtuse (> PI / 2), since the sinusoidal function is symmetrical around PI / 2 on the interval [0, PI]. To check if the angle is obtuse, you also need to use the cosine theorem, although you do not need to calculate the cosine itself: if | AB | ^ 2 + | AC | ^ 2> | BC | ^ 2, the angle at A is obtuse.
I think this sums it up.
zvrba source share