Service Bus for Window Server 1.1 supports messages of up to 50 MB (unlike its Azure counterpart, which supports only 64 KB). If this size isn't enough, there's a pattern called Splitter that allows to overcome this limitation. The Service Bus is owned by the same team that created MSMQ and offers many similar features, as well as some new ones. It requires a SQL Server instance for its storage.
There are also other options such as ActiveMQ and RabbitMQ, both of which are open source.
All of the above have WCF bindings, as well as a standalone API.