No. The whole point of quirks mode is that it's a compatibility mode for IE5. This means that in addition to changing the layout mode, it also switches off most of the browser features that have been invented since IE5.
Therefore the blunt answer is no, you cannot mix Quirks mode and HTML5. It just can't happen.
However there is some good news for you: switching from quirks mode to standards mode is actually easier than it looks at first glance.
You don't have to go through your whole site changing all the CSS to suit the different box model, because standards mode does have a CSS feature that allows you to use the quirks mode box model while still remaining in standards mode.
Simply add the following to the top of your CSS code:
* {box-sizing:border-box;}
This will change all your elements to use the quirks mode box model, but your page will still be in standards mode.
This should sort out most (if not all) of the layout issues you've been having.
Hope that helps.