While it may be tempting to think that the "-12" somehow corresponds to a POSIX error number (ENOMEM being 12 would be a somewhat fitting candidate), I fear that the "-12" is just a value hard-coded into R3 without a particular meaning. In the TCP port's "actor" code we find the following line:
if (OS_DO_DEVICE(sock, RDC_OPEN)) Trap_Port(RE_CANNOT_OPEN, port, -12);
So a "cannot open" error is caused, when, well, the port cannot be opened. Along is passed the magic number "-12".