I was complicating this way to much in my mind. I added one line to the $args array and voila!
$args=array(
'post_type' => $post_type,
"$tax" => $tax_term->slug,
'portfolios' => 'portfolio-two',
'post_status' => 'publish',
'posts_per_page' => -1,
'caller_get_posts'=> 1
);
Now I get list of all posts from post-type music that are marked portfolio-two (taxonomy). They get listed up under the genre taxonomy they are a part of.