Yeah it is the 64 bit IEDriverServer. Something about IE's content handling and other technical jargon. But I deleted the 64 bit driver and downloaded the 32 bit one and now it is working just as well as chrome and firefox. Here is a link to all the files.
http://selenium-release.storage.googleapis.com/index.html
Also note for work reasons I am using IE10 but its something to try if your on IE11