Why is this a requirement, does anyone know?
My understanding (based on a really old forum post by an Intuit developer) is that the way the SDK works is by using a GUI message pump Windows COM call to push data to QuickBooks.
No GUI present = no message pump = no connection to QuickBooks. Thus, you need the GUI components of QB in place.
Is there another way to directly access the data and bypass the massive overhead of the QuickBooks application?
No.
If it's a problem, consider batching your requests, queueing them up, and sending more than one at a time in a batch. This is what the Web Connector (and most other QuickBooks integrated apps) do to avoid this issue. It's very, very rare that an application truly needs real-time connections to QuickBooks (and be careful if you think you do - QuickBooks is not a great candidate for real-time access to data - there's a lot of things that can lock you out of QuickBooks so you have to be careful if you're building an app that assumes you'll always be able to connect to it).