If you have all the pixel data in memory (which I would suggest that you do this, but this is the main anchor point) and there are only two different colors, all you have to do is run a horizontal scan line to find the point , in which RGB changes from color X to color Y (note that you may need to run this scan line several times, but in any case it is no worse than O (height)).
A simple graph traversal (BFS or DFS) will continue along this line (you only need 3 points, and then you can form a geometric line with the equation a * x + b * y + c = 0 (assuming that this is not a curve)) . Repeat this vertical scan (again, in the worst case, this is O (width)). Find 3 points, and then you will have two lines with d * x + e * y + f = 0. Using a little comp. geom, the intersection of these two lines will give you your point.
source share