As I see it, the security is not a problem, as long as you accept connections from well known clients. A packet header with some indicators at the begging of the conversation with a random generated password (based on time/date for example) would be enough to acknowledge the client.
At the firewall you should allow those ports to get through.
Also, if the interchanged data is confidential consider encrypting it to avoid someone sniff the data.
There is also a problem if someone wants to open a lot of connections simultaneously, at the server allows a reasonably numbers of open connections at a time, so that the server won't get exhausted.