As system
also escapes spaces in arguments, your system call with -oX -
will effectively be called as
nmap "-sn" "-oX -" "example.com"
with the space being part of a single argument. It will thus not be considered a valid argument for nmap. To fix this, you actually have to pass it as two arguments. Here, the space in the single argument will not be escaped:
system('nmap', '-sn', '-oX', '-', hostname)