x264 will encode video using the h.264 codec and only supports a handful for containers. It will not encode audio, and will not encode video using other codecs such as vp8 (webm) theora, etc
mencoder will encode both audio and video and supports many more codecs and containers. Internally mencoder will x264 to encode h.264 video.