I am now thinking about calculating the position of the scroll in the contact list and render on the displayList only what's on screen but I'm not sure how to handle this.
That's the correct way to do it, it's a form of object pooling and I think it's also known as layout virtualization. I don't know how to do it in classic AS, but I've been using the Starling framework (gpu rendered display list), and the components lib there (known as Feathers), has such a list, you may wish to check its implementation. Here's a demo (check the List), I've tested this with thousands of items and it works perfectly: Feathers Component Explorer
But in short the idea is to create visual components equal to the maximum that can be seen at the same time. Then, whenever the list moves you must check which are the visible indexes. When they change, which happens when an item becomes invisible - for example going off the top, you move it to the bottom and reuse it with new data, corresponding to its index.