You could try something like this. Modify your function to return only 'horizontal' class if image is landscape:
function mytheme_vertical_check() {
$thumb_id = get_post_thumbnail_id();
$image_data = wp_get_attachment_image_src( $thumb_id , 'tp-thumbnail' );
$width = $image_data[1];
$height = $image_data[2];
if ( $width > $height ) {
return 'horizontal';
}
}
And inside the loop call it to add 'horizontal' class to a div of your choice:
<?php if ( has_post_thumbnail() ) : ?>
<?php printf( '<div class="masonry-thumbnail %s">', mytheme_vertical_check() );?>
<?php the_post_thumbnail( 'large' ); ?>
</div>
<?php endif; ?>