1) For the unsigned option:
First you need to load x into a register to square it:
mov eax,0
mov al, x
mul al
this essentially did this: AX = x*x, and AX is word size, since the upper part of EAX is all zeros, then essentially EAX = x*x, and EAX is already double word size. Now we need to subtract y and add 2:
mov ebx, y
sub eax, ebx
add eax, 2
and there you have it! the answer is inside EAX.
2) For the signed option:
We will basically do the same, however this time we will make use of the CWDE instruction which will extend the value inside AX into EAX, while keeping the sign information correct, so:
mov al, x
imul ax
cwde
mov ebx, y
sub eax, ebx
add eax, 2