You need to include the concluding Enter in the mapping, just as you'd type it.
:map <M-o> :VoomToggle markdown<CR>
Vim uses a special notation, like in the mapping keys; see :help key-notation
.
Also, you should use :noremap
; it makes the mapping immune to remapping and recursion.