Have a look at this page http://rosettacode.org/wiki/Y_combinator.
Applied to your case it gives:
1> Y = fun(M) -> (fun(X) -> X(X) end)(fun (F) -> M(fun(A) -> (F(F))(A) end) end) end.
#Fun<erl_eval.6.82930912>
2> Mul = fun (F) -> fun ({X,0}) -> 0; ({X,N}) -> X + F({X,N-1}) end end.
#Fun<erl_eval.6.82930912>
3> (Y(Mul))({5,4}).
20
4>
I must confess it is a bit complex for me...