Ok, after trying around a while i found out what causes this behaviour. The MSDN article of the UIElement.RenderTransform property says:
A render transform does not regenerate layout size or render size information. Render transforms are typically intended for animating or applying a temporary effect to an element. For example, the element might zoom when focused or moused over, or might jitter on load to draw the eye to that part of the user interface (UI).
That means, that the polygon itself, respectively the points of the polygon, isn't transformed. When applying the ScaleTransform the points of the polygon do not change and will remain as they were set in the xaml. Nevertheless the transformed polygon will react on mouse events, but the mouse position is mapped to the original bounds.
Example: Let's take the polygon, defined above in the xaml and scale it by factor 1.5. The polygon is displayed scaled and it seems that its points has the values (150,150 150,450 450,450 450,150) now. But internally it still has the old values (100,100 100,300 300,300 300,100). If you catch the position of the mouse pointer at the scaled top right corner, it will be (300,100) instead of (450,150).
Using the mouse position as center for the scaling causes the shift of the scaled polygon.
A solution for this is to scale the points of the polygon itself, instead of using ScaleTransform:
void Scale(double factor, double centerX, double centerY)
{
PointCollection pc = new PointCollection();
foreach (Point p in polygon.Points)
{
Point q = new Point();
q.X = p.X;
q.Y = p.Y;
q.X -= centerX; // translate
q.Y -= centerY;
q.X *= factor; // scale
q.Y *= factor;
q.X += centerX; // translate back
q.Y += centerY;
pc.Add(q);
}
polygon.Points = pc; // polygon is defined in xaml
}
I'm not sure if this is the right way to do it, but it works for me.