Everything depends on what the application using such device expects.
If /dev/ttyTEST
is to behave like a real serial device and respond properly to all ioctl's that set its speed etc., then this can't be done from userspace. It wouldn't be too hard to implement in the kernel space, though.
If /dev/ttyTEST
only needs to be a tty, then provide a pseudo tty.
If /dev/ttyTEST
is merely to be something another application can write to and read from then socketpair()
does it.
If you have control over the application's code, then you can have it check whether the device is a socket pair or a real character device, and ignore the failures of the serial-port-specific APIs on a socket.