Do you just need to check for self-intersections or find all of them? The latter is more complicated than O(N log N) , since you can have O(n^2) intersections with segments n .
If you only need to find out if there are self-intersections or to find a small number of them, then look here . This article seems to require only what you need, especially in the planarization section of the polygon. I doubt that the described algorithm will be simple or appropriate for any problem of a reasonable size. But such an algorithm exists. Disclaimer: I did not try to work on the document and understand it.
source share