There are no decimals in batch arithmetic. All the calcs are with 32 bit integers (31 bit numbers plus a sign bit), with data values in the range -2147483648 to 2147483647. So what you need is
set /a "memoryToUse=%freeMemory% * 9 / 100"
Of course, limited to the indicated range of values.