JMS is Java Message Service and since Domino is an application server that supports Java, you should be able to use JMS with Domino.
AMQP is Advanced Message Queuing Protocol and is a protocol that requires a MQ implementation in order to be used. If you use an implementation of AMQP that supports Java, then you can use this with Domino too.
Here's an article that shows how to create an agent that uses JMS: http://www.ibm.com/developerworks/lotus/library/domino-jms/