An advice first: try to keep your code as simple as possible. Your code can reduce to this (that surely works)
sortedunion(A, B, S) :-
append(A, B, C),
sort(C, S).
but of course it's instructive to attempt to solve by yourself. Anyway, try to avoid useless complications.
sortedunion(A, B, S) :-
undup(A, N),
undup(B, M),
unite(N, M, S).
it's equivalent to your code, just simpler, because A = [H|T]
and so on.
Then test undup/2:
1 ?- undup([1,1,1,2,3,4,4,5],L).
L = [_G2760, _G2808, _G2847, _G2877, _G2889] ;
false.
Clearly, not what you expect. The culprit should that anon var. Indeed, this works:
undup([], []).
undup([X|Xs], [X|B]) :- remove(X,Xs,K), undup(K, B).
2 ?- undup([1,1,1,2,3,4,4,5],L).
L = [1, 2, 3, 4, 5] ;
false.
Now, unite/3. First of all, is/2 is abused. It introduces arithmetic, then plain unification suffices here: X = S
.
Then the base cases are hardcoded to work where lists' length differs at most by 1. Again, simpler code should work better:
unite([], [], []).
unite( X, [], X).
unite([], X, X).
...
Also, note the first clause is useless, being already covered by (both) second and third clauses.