One possibility is that you are terminating due to the SIGCHLD signal not being ignored and causing your program to abort.
signal(SIGCHLD, SIG_IGN);
Another is, as you suspect something actively closing the X session. Just closing the socket itself should not matter but are you using a library that registers an atexit call it could cause an issue.
Since from your snippet, it looks like you don't actually care about the stdout of the xterm, a better way to do it would be to actuall close fd's 0,1,2. Also since it looks like you don't need to do anything in the child process after xterm terminates you can use 'exec' rather than 'popen' to fully replace the child process with that of the xterm including any cleanup handlers that were left around. Though, I am not sure how pruned your snippet is from what you want to do as obviously the call to 'gets' is not what you want.
to make sure the X connection is closed, you can set its close on exec flag with the following. (this will work on POSIX systems where the x connection number is the fd of the server socket)
fcntl(XConnectionNumber(display), F_SETFD, fcntl(XConnectionNumber(display), F_GETFD) | FD_CLOEXEC);
Also note that 'popen' itself forks in the background in addition to your fork, I think you probably want to do an execvp there then use waitpid(... , WNOHANG) to check for the childs termination in your main X11 loop if you care to know when it exited.