The documentation link in your question is actually to the HTML5 player and not the Flash player. The ability to mute a video from a custom button is still available in the Flash version and the API docs can be found here - flowplayer JavaScript API
mute()
is a method available on the player, and also unmute()
and can be used like this:
Usage
$f('player').mute();
$f('player').unmute();