Let’s look at just the integer representations. You have two 16-bit integers, x and y, and you form their 16-bit two’s complements. However, you keep these 16-bit complements in 32-bit objects. In 32 bits, what you have is 65536–x and 65536–y. (For example, you started with 0xa78, complemented it to make 0xfffff588, and discarded bits to get 0xf588. That equals 0x10000-0xa78.)
When you multiply these, the result is 65536•65536 – 65536•x – 65536•y + x•y.
The 65536•65536 is 232, so it vanishes because unsigned 32-bit arithmetic is performed modulo 232. You are left with – 65536•x – 65536•y + x•y.
Now you can see the problem: x•y is the product of two 16-bit values, so it flows into the high 16 bits of the 32 bits. Up there, you still have – 65536•x – 65536•y, which you do not want.
An easy way to do this is multiplication to keep all 32 bits of the complement. E.g., when you took the two’s complement of 0xa78, you got 0xfffff588. Then you discarded the high bits, keeping only 0xf588. If you do not do that, you will multiply 0xfffff588 by 0xffffff4b, and the product will be 0x766d8 which, when shifted for the fraction, will be 0x766, which is the result you want.
If the high bits are lost because you stored the two’s complement into a 16-bit object, then simply restore them when you reload the object, by extending the sign bit. That is, take bit 15 and repeat it in bits 16 to 31. An easy way to do this is to load the 16-bit object into a 16-bit signed integer, then convert the 16-bit signed integer to an unsigned 32-bit integer.