MP4 type isn't supported on Firefox! It's only supported in Safari 3.0+, Google Chrome 5.0+ and IE 9.0+! For Firefox you'll need .ogg file or .webm video files as sources! Here is an image containing all supported video formats in HTML 5:
And for audio support see this pic:
UPDATE:
Firefox now supports MP4 H.264 (AAC or MP3) https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats
Note: MP4s encoded with a high profile will not run on lower end hardware, such as low end Firefox OS phones.