I wrote the answer to this question in the “Mathematical Stack Exchange” section and included the image that I created using the concept proof concept. The approach used for this can work for many practical applications, so I will describe it here in more detail.
Take your shape and zoom it using a polygon. Define runs of consecutive vertices with an internal angle> 180 °. For each such run, iterate over all possible tangent lines. A tangent line for a run is a line connecting two consecutive vertices, at least one of which lies within the range. This means that the first line is determined by the last vertex before starting and the first vertex of the run. Take the inner directional half of the plane defined by this line and cross it with your shape, then calculate the area and compare it with the best solution found so far.
In a simplified approach, you simply set up a recursive schema to try out all possible string combinations. this means the time complexity O ( n m ), where n is the number of vertices and m is the number of runs. In a more complex approach, you can use the fact that a line that does not intersect with any other line inside the form can be selected independently of the others. The same is true for groups of lines that do not intersect with each other within the form. Some line options will be completely discarded in another way, so the choice made for this launch becomes irrelevant. Thus, there are many opportunities for smart algorithms, depending on how much effort you can invest and what kind of performance you need.
If your input is a polygon, then the line touching the non-convex vertex does not have to coincide with the incoming or outgoing edges of the polygon, but can be arbitrarily rotated between these two limits. In this case, the above approach may provide a non-optimal solution. But since you noted in the commentary that we can consider “non-polygonal” forms, which I perceive as “smooth”. In this case, you have a well-defined tangent at each point, and each tangent will be close enough to the edge of the polygonal approximation.
Unlike what I thought initially, the above should work for figures with holes, since the boundary of a convex hole will lead to a non-convex path of the figure itself. Thus, the launch will lead to the fact that you are exploring all possible ways to cut holes. SImilar for non-convex holes: matching runs also allow you to cut them without losing convex solutions.
As applied to your example, the algorithm gives the following result:
The non-convex runs of the peaks are colored red, the best set of lines in blue and the resulting area in green. The polygon behind it has 269 vertices. The implementation was performed in Java, without regard to performance, the brute force of a recursive search for all possible combinations and some assumptions that work for this input, but may not work at all.