Because the regex is not anchored in any way, it is free to be as loose as it likes.
In this case, let's look at how your string can be broken down. the square brackets indicate the attempted match.
... </p>[ </li>] // Fails, lookbehind assertion denies match
... </p> [ </li>] // Succeeds, lookbehind sees a space, not </p>
So you see the match succeeds simply by matching one less space, which is why you see a space between the two </p>
in the result.
There's no easy fix for this in Regex. THE PONY HE COMES. So instead try using a parser.
$dom = new DOMDocument();
$dom->loadHTML($html);
$lis = $dom->getElementsByTagName('li');
foreach($lis as $li) {
if( !$li->getElementsByTagName('p')->length) {
$p = $dom->createElement("p");
while($li->firstChild) $p->appendChild($li->firstChild);
$li->appendChild($p);
}
}
$output = $dom->saveHTML($dom->getElementsByTagName('body')->item(0));
$output = substr($output,strlen("<body>"),-strlen("</body>")); // strip body tag