The result of the comparison function should return a negative number, 0, or a positive number. You are returning only 0 or 1.
Your comparison function should return something like this:
return px->x < py->x ? -1 : px->x == py->x ? 0 : 1;
or terser but a little more opaque:
return px->x - py->x;
See qsort reference. It's technically a C++ reference page, but the explanation is good for C as well.
ADDENDUM
I forgot to explain what happened, sorry! Your comparison function does the following.
Whenever
px->x < py->x
, your function returned 1, making it think thepx
tuple was greater than thepy
tuple, when in fact is what not. (You probably wanted to return a negative value in this case.)Whenever
px->x >= py->x
, your function returned 0, makingqsort
think that the two values were equal, when in fact they may or may not have been.
So qsort
is just blindly partitioning and swapping elements according to what your comparison function was telling it the order was. As your function was giving back only "equal" (0) or "greater" (1), and never "less", the final result turned out to be rather scrambled.