could quirks mode be to blame?
If you're in compatibility mode (ie doc mode is 10, 9, 8 or 7) then by definition you're not in quirks mode (which is doc mode 5), so the short answer to that part of your question is No, it isn't quirks mode.
However, compatibility mode can cause itself cause issues. The whole point of compatibility mode is that the browser pretends to be an older version of itself. So in doc mode 8, you have IE11 pretending to be IE8.
That pretence isn't hugely accurate (so don't think that you're seeing your site the same as a real IE8 user would see it), but it does mean that IE11 will switch off various browser features in order to make itself work more like IE8.
Therefore, if your code is relying on a browser feature that was introduced after IE9, then that would indeed probably break your site in doc modes 9 and below. But without seeing any code, it is absolutely impossible for me to speculate any further about which particular aspect of your code would be causing this.
The only thing I can suggest is that you might get some clues by checking the console for error messages, but in reality if you want help, then you're going to need to need to swallow that "I can't share code" issue.