Wrong tessellation of self-intersecting contour
I tested a little your library and I created a case which is incorrectly tessellated. I attached a test file - it contains the problematic contour. First line contains number of vertices and then followed vertices (x, y, z values split by spaces). contour3.txt
I use the LibTess2 library with doubles instead of floats (TESSreal type). The problematic intersection is between edge (2-1) and edge (75-76) (numbers are the indices of vertices). I tried also to debug the the code in sweep.c but I didn't find exactly where the problem is. If all vertices are slightly rotated around z-axis (like by 5 degrees) than the tessellation is correct.
note: option TESS_CONSTRAINED_DELAUNAY_TRIANGULATION changes the output triangles, but it produces same filled area
The wrong tessellation:

See this comment describing a much simpler error case. https://github.com/memononen/libtess2/issues/37#issuecomment-1958287801
I uploaded some source code that doesn't seem to exhibit that bug.