After considerable investigation, I came across this issue in the microsoft bug tracker:
The description:
Create a simple data class that implements INotifyPropertyChanged, change all the items' color every 5 secs, but some items have different background color with different zoom levels. This issue was reported from Silverlight forum: http://forums.silverlight.net/p/262082/654612.aspx/1?Re+PivotViewer+SL5+Dynamic+Collection+to+change+item+background+color
So I guess that explains what the problem was. Just waiting for a fix now.