I would go with a direct connection to your own server, mainly because as you say, the GCM service wasn't really designed for this and Google hold the right to throttle the connection and delay the GCM notifications if they think you are using too much bandwidth. I've experienced this myself when conducting stress tests - the notifications start slowing down and eventually the gmail accounts that the devices where using were banned.
The general pattern for GCM should be something similar to:
Something happens on the server that the devices need to know about.
A GCM notification is sent to the device with a small payload, say CommandNumber = 1 for example.
The device listens to the notification, and because the CommandNumber == 1 it responds in a certain way, which could be opening a socket to your server in this case.