You just need to escape the space. Use this instead.
:set foldtext='\ '.foldtext()
The space is causing vim to think you want to set foldtext to '
and then '.foldtext()
is the next argument to set. However this isn't what you want and the reason the error message is
E518: Unknown option: '.foldtext()
Escaping the space tell vim that foldtext='\ '.foldtext()
is one argument instead of two.