(?!\\)\\
Is a bit of a contradiction. "The next character must not be a \
, then match a \
as the next character."
Since JavaScript doesn't currently support look-behinds, you can capture a negative character group and just put it back into the string:
replace(/(^|[^\\])\\\(/g, '$1~~\\(')
The ^|
allows it to also match at the beginning of the string since the [^\\]
alone would require a leading character.