This is referred to as Port Unification. There is a good example of it here, although it demonstrates switching between TCP and HTTP (with SSL and/or GZip detection), and not websockets, but the principles are the same.
Basically, you will read in the first 5 bytes to sniff the protocol (more or less as you did) and when the protocol is identified, modify the handlers in the pipeline accordingly.
Since you need to initiate a websocket through HTTP anyway, the example should work for you if you add the websocket upgrade procedure as outlined in this example.