After doing lots of efforts, I came to know that I was making a silly mistake by putting a data grid inside a scroll bar, which disables the by default visualization(UI and Data), and It was trying to load all the objects in the grid regardless of their need,
So, good note:
Never ever place scroll bar outside a **grid.
<ScrollViewer>
<DataGrid>
....
</DataGrid>
</ScrollViewer>**
Never do this.