You can use any TCP or UDP protocol to talk between App Engine and Compute Engine. How? Socket API.
For Java: https://developers.google.com/appengine/docs/java/sockets/overview
For Python: https://developers.google.com/appengine/docs/python/sockets/overview
You would still need to implement the protocol over TCP or UDP, or find a library that does it for you.