First we take care of the trivial case where the angle is zero or a multiple of pi / 2. Then the largest rectangle matches the original rectangle.
In general, the inner rectangle will have 3 points on the borders of the outer rectangle. If this is not the case, then it can be moved so that one vertex is at the bottom and one vertex is on the left. You can then enlarge the inner rectangle until one of the two remaining vertices reaches the border.
We call the sides of the outer rectangle R1 and R2. Without loss of generality, we can assume that R1 <= R2. If we call the sides of the inner rectangle H and W, then we have that
H cos a + W sin a <= R1 H sin a + W cos a <= R2
Since there are at least 3 points on the boundaries, at least one of these inequalities should actually be equality. Let me use the first one. It is easy to see that:
W = (R1 - H cos a) / sin a
and therefore the area
A = HW = H (R1 - H cos a) / sin a
We can take the derivative with respect to. H and require that it be 0:
dA/dH = ((R1 - H cos a) - H cos a) / sin a
Solving for H and using the expression for W above, we get that:
H = R1 / (2 cos a) W = R1 / (2 sin a)
Substituting this into the second inequality, it becomes after some manipulations
R1 (tan a + 1/tan a) / 2 <= R2
The coefficient on the left side is always not less than 1. If the inequality holds, then we have a solution. If this is not satisfied, then the solution is one that satisfies both inequalities as equalities. In other words: it is a rectangle that touches all four sides of the outer rectangle. This is a linear system with two unknowns that can be easily solved:
H = (R2 cos a - R1 sin a) / cos 2a W = (R1 cos a - R2 sin a) / cos 2a
In terms of the initial coordinates, we obtain:
x1 = x4 = W sin a cos a y1 = y2 = R2 sin a - W sin^2 a x2 = x3 = x1 + H y3 = y4 = y2 + W