First of all, val is a function, not a variable, requiring $('#producttextarea').val('');
to change the contents.
The Jquery dialog is actually something which exists on the page as a hidden div of sorts. If you want to clear out that, you would need to, at the time the dialog is opened, clear out the contents of val. Alnitak's solution works well for this, or you could put it in the click handler, as such:
jQuery( "#popup" ).click(function() {
jQuery( "#dialog-form" ).dialog( "open" );
jQuery( "#producttextarea" ).val('');
});