$recordset->open
does not return a RecordSet, you should not be assigning the return value as you are doing.
Try changing:
$recordset = $recordset->Open($sql, $conn);
To
$recordset->Open($sql, $conn);
In addition, your ported code significantly deviates from structure of your VBScript version. Here is my attempt at porting your code, which works for me.
$conn = new COM("ADODB.Connection") or die("Cannot start ADO");
$recordset = new COM("ADODB.Recordset");
$conn->Open("Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
// your query doesn't work for me in VBSCRIPT or PHP. this one does.
$recordset->Open("SELECT Top 5 System.ItemPathDisplay FROM SYSTEMINDEX", $conn);
if(!$recordset->EOF) $recordset->MoveFirst();
while(!$recordset->EOF) {
echo $recordset->Fields->Item("System.ItemPathDisplay")->Value . "\n";
$recordset->MoveNext();
}