I'd guess that the returned bitmap is simply too large for memory, given that you get an out of memory error.
You should try using Ghostscript directly from the command line for 2 reasons, firstly you will be able to see if there is a real error message about the xref, which would indicate your PDF file is damaged, secondly you might reasonably just run a shell command to invoke GhostScript to render the PDF directly to JPEG rather than going through an in-memory bitmap. Its probably faster apart from anything else.
gswin32c -sDEVICE=jpeg -o out.jpg input.pdf