var contains = $("ul#playlist li:contains()");
supposed to be
var contains = $("ul#playlist li:contains(" + song + ")");
Otherwise you can have a array and push the songs when added to the playlist. And check the array first before adding it to the playlist.
And your condition should check for the Length of contains
if (contains> 0) {
supposed to be
if (contains.length > 0) {