Algorithm for determining linear behavior in a dataset

Question

I posted a question about the Algorithm to make a polynomial fit to a part of a dataset some time ago and got some suggestions to do what I wanted. But now I am facing another problem, I am trying to apply the ideas proposed in the answers. My goal was to find the best linear fit for a dataset in which only part of it was linear.

Here is an example of what I should do:

We have these two data sets, and I have to make a linear trend of the linear part of the data, which is to the left of the dashed line. In red, we have a perfect data set that has a linear part from the beginning to the dashed line. In blue, we have a “problematic” dataset that has a plateau. The bold part is the part that I should use to fit the data linearly.

My problem is that I tried to do as mentioned in the question above: I found the second-order derivative of the smoothed data and looked when it was not “close enough” to 0. But here are my results for a problematic dataset (first image ) and an ideal data set (second image):

(Sorry for the quality, I don’t know why it is so blurry) In both images I built a first-order derivative and red - a second-order derivative. In the first image, we see peaks in the values ​​of the second derivative. But the problem is that the peaks are not very "high", which makes it difficult to establish a threshold that will tell if the set is linear or not ... On the contrary, the peak of the first derivative is quite high, which makes it easy to see visually.

I thought that calculating the average of the values ​​of the first derivative and see if the value is too much different from the average is enough ... But when I take the average of the values ​​of the first derivative, to see where the values ​​differ from the average, there is a sort of bias due to peak.

How to remove this offset in order to take only the average value of the data on the right (the data to the left of the gap, which are visible in image 1, can be non-linear or linear, but have a different value from the value on the right!) Is the peak effective?

Answer 1

The mean operator (as you noticed) is very sensitive to outliers (peaks). You can use more reliable estimates, such as median or x-percentile values ​​(which should be more suitable for your case).