Using the Web Font Loader javascript instead of the default method of adding fonts works in IE11 for me. https://developers.google.com/fonts/docs/webfont_loader.
It also had the advantage of having default font text showing on slow connections until font downloads, at which point it will switch to google font. Otherwise you get no text at all for tens of seconds.
Important note: You should choose your fall-back fonts to those with similar size / kerning / leading etc to minimise screen re-draw and content jumping issues that affect UX.