VMR9s/EVRs are backed by limited resource, you are hitting the limit. There is no distinctive error code or otherwise well known/reliable way to detect that you are in trouble.
Your workaround is to either implement custom allocator/presenter to manage surfaces, or use VMR7 instead.
See also:
- StressEvr: So, how many EVRs you can do? with a small tool/testbed to see how many instances you can create, VMR7/EVR, at specific resolutions and pixel formats