The solution is that I've added another div which contains the text. then I take the height from the tr element and put it as the width of the text div. This will calculate the correct text overflow length.
here the corrected version http://jsfiddle.net/rpQew/
the new css class:
.overflow{
top:5px;
position:relative;
display:inline-block;
width: 150px;
text-align: left;
padding-left: 5px;
padding-right: 5px;
overflow:hidden;
text-overflow:ellipsis;
white-space:nowrap;
}
the table:
<table border="1">
<tr id="tableRow">
<th class="positionFix"><div class="rotate"><div class="overflow">item 1 Test text text-overflow test</div></div></th>
<th class="positionFix"><div class="rotate"><div class="overflow">item 2 more text foo bar faz</div></div></th>
<th class="positionFix"><div class="rotate"><div class="overflow">item 3 foo bar foo bar foo bar foo bar foo</div></div></th>
</tr>
<tr>
<td>entry 1</td>
<td>entry 2</td>
<td>entry 3</td>
</tr>
</table>
and the js:
var rowHeight = $('#tableRow').height();
$('.overflow').width(rowHeight+'px');