I'm currently working on a project that is building a java-based desktop application to interface with a website. We want to incorporate IM capabilities, so we decided to use XMPP.
The problem is our application has other features, and anyone using another client to connect to our XMPP server will cause problems with our website (e.g. our client will be able to send our messages with a certain message type that the user won't be able to use, but with another client they could send those message types).
Is there anyone to either allow only our client to access the XMPP server or prevent other clients from using certain features? I know this is against the idea of open standards, but we don't want to build a proprietary IM solution from scratch.
You are building a proprietary solution, it just might not be completely from scratch, and that's not necessarily a bad thing. But please don't call it a XMPP service unless you are going to support XMPP clients. You will get the same reaction as you get with a "web site" that requires your proprietary browser.
For features that can be negotiated, look at Feature Negotiation and you might be able to get away with saying your server doesn't have a specific feature to other clients, but secretly supporting it in your own. That won't actually block something from being attempted, so it's pretty poor solution.
You can get instant messaging capabilities without building a desktop application (with all of the platform support headaches that entails). Consider Orbited which can give you the instant messaging interactivity your looking for and would make it much easier to integrate on the server.
And just as a side point, there shouldn't be any messages that cause problems with your web site, any more than there could be a URL that causes it problems, or a query combination, etc.
Not sure of all your requirements, but it sounds like you could probably use the pubsub or pep features of XMPP. These are extensions to XMPP that allow you to create specialized payloads that can be accessed on a user to user level (Personal Eventing Protocol XEP-0163). If it is simply a general notification to everyone who is interested, then PubSub (XEP-0060) may be what you want.
These protocols allow for securing access to the pubsub nodes and will not get affected by the standard chat messages, as they are a different protocol.