I found a way, though it has some side effects as it changes the global variable buffer-face-mode-face
For example, I want to set current buffer, I can eval this function definition and run it
(defun my-buffer-face-mode-serif ()
"Sets a fixed width (monospace) font in current buffer"
(interactive)
(setq buffer-face-mode-face '(:family "Times New Roman"))
(buffer-face-mode))