Typically with Saxon the space occupied by the source document as a tree is around 5 times the size of the lexical XML. The ratio can vary, of course, depending on the complexity of the document. So for 550Mb, allocating 3Gb should be enough. Use the -Xmx option when starting up Java to allocate enough heap space.
You're close to the limits of what can be done using a conventional in-memory transformation, so you might like to look at the options for doing a streamed transformation. For information on this, see http://saxonica.com/documentation/#!sourcedocs/streaming