According to QTP11's Product Availability Matrix (which you can only see if you have an HP passport and I suppose you don't have considering that you're using a trial version). Windows 8 and 8.1 are not supported with QTP11. This isn't surprising since QTP 11 was released almost two years before Windows 8.
I know for a fact that QTP11 doesn't support IE11 which is the earliest version of IE supported on windows 8.1.
Since you're using a trial version in any case why not download the latest version (UFT 12)?
QTP was united with Service Test and is now called Unified Functional Testing (UFT) and version 12 does support Windows 8.1.