Hmmm. You're on the right track, but I think you're wandering a bit.
A ListView is a view object, not a BaseAdapter. As far as I can tell from your code, you don't need a custom adapter for your ListView.
What you should do is bind a regular CursorAdapter to your ListView, load the Contacts data you want from the Contacts Provider using a CursorLoader, then move the resulting Cursor to the CursorAdapter.
You can find the instructions for doing this in this Android training class: Loading Data in the Background.
What do you want in your custom ListView that you can't get from ListView itself?