You have a case of competing event handlers, and the last one (the one that sets the height to 5.7em) wins. The event handlers actually stack, so you keep adding additional copies of the one that closes the menu.
Try maintaining the open/closed state independently. Something like this (jsfiddle):
$(document).ready(function () {
$('#nav').data('open', 0).mouseup(function () {
if ($(this).data('open') == 0) {
$(this).css('height', '33em').data('open', 1);
} else {
$(this).css('height', '5.7em').data('open', 0);
}
});
});
Or better yet, move the CSS details to a CSS class:
#nav { height: 5.7em; }
.open { height: 33em !important; }
...and just toggle that class on and off (jsfiddle):
$(document).ready(function () {
$('#nav').mouseup(function () {
$(this).toggleClass('open');
});
});