There is one parameter missing from your definition. For this reason, your (if ...)
form is actually interpreted as the keymap
parameter.
Try this:
(define-minor-mode
myspace-mode
"some doc"
nil
nil
nil
(if myspace-mode
(whitespace-mode 1)
(whitespace-mode -1)))