So Finally I did manage to find a way to do it. First approach was as @Murkaeus suggested, Using UserControl and ManipulationDelta event handler. However for some reason the manipulationDelta event was triggered only for 2 finger gestures (Zoom, Pinch..etc), I have no clue why. And after some trying I had to give up on this.
The next approach was using Listbox itself. The source of the Listbox was set as the List( of Models objects) that I create after reading the contact information from phone. The height and color of the listbox item was bound to a property in my model named "scaleLevel" and was accordingly converted by implementing IValueConvertors to give predefined color and height values for different scale levels.
I created an attached property for the scrollviewer vertical offset like mentioned [here] (http://www.scottlogic.co.uk/blog/colin/2010/07/exposing-and-binding-to-a-silverlight-scrollviewer%E2%80%99s-scrollbars/)! This event is triggered on change of the vertical offset and every time there is a scroll, I find out which listbox item to be enlarged and which reduced based current vertical offset.
Once I have this information, I change the property ("scaleLevel") of the affected items in the List (of Model) (which is bound to listbox height and color). This change is updated in UI using the INotifyPropertyChanged Event.
I have no idea if this the best way of doing it , but it works well and there is no considerable in updating the UI despite the processing involved.
I would like to hear your opinion about the implementation and any other solution which you feel will work better.