I've seen this same behavior with Ruby and an Excel library: a non-trivial amount of time to open a large file, where large is > 500KB.
I think the cause is two things:
1) an xlsx file is zip compressed, so it must first be un-compressed
2) an xlsx file is a series of XML files, which all must be parsed.
#1 can be a small hit, but most likely it pales in comparison to #2. I believe its the XML parsing that is the real culprit. In addition, the XML parser is a DOM-based parser, so the whole XML DOM must be parsed and loaded into memory.
I don't think there is really anything you can do to speed this up. A large xlsx file contains a lot of XML which must be parsed and loaded into memory.