YouTube, Netflix and other app developers and hardware OEM's are working on the DIAL protocol: http://www.dial-multiscreen.org/home The site contains the protocol spec and some C code to simulate server and clients. It is likely that the current YouTube app is using this, but YouTube has not made any information about their protocol implementation available to developers.
Google TV devices are DIAL servers. You can discover GTV devices on your network using DIAL. However, what is missing is how to launch and control individual apps. It looks like GTV devices don't have the full implementation yet. Also, documentation needs to be provided by each of the app developers to understand what commands/protocols they support once they are discovered and launched. DIAL is more generic than Anymote and is likely how GTV devices will work in the future, especially since the protocol is designed to be platform and device independent.