Is it acceptably performant to import data into Magento via the ORM, or is straight SQL the only option?

Question

Firstly, my experience with Magento is very small, and my level of knowledge is barely deeper than the surface. I've been reading, but I'm still really uncomfortable with what I know.

• I've tried MAGMI - out of the box it only imports products, but I need quotes, orders, addresses, customers, and different price lists

• I've attempted to create an extension for MAGMI - but I'm not comfortable enough with, nor do I know enough about the database structure to write queries for it.

• I've written a module that attempts to import customer information using the SOAP interface, but it's absolutely and horrendously slow. (~5 minutes to connect and import a single customer with complete information)

I need to write a module that imports several things from a custom XML file.

I don't have any control over the data that's coming in beyond having the XML file itself - I'd prefer to just import it as it is instead of converting it to several CSVs.

To avoid having to write to the database, I tried to write a process to import customer information using the SOAP interface but it was painfully slow - I'm talking 25-30 seconds just to connect to the interface, and then another 30 seconds or so to actually import a single customer, without their address or any other information.

I'm either doing something very wrong - or I'm not using that system as intended, but in any case, I'm going to extend my module to import products using either Magento's ORM, or write directly into the database.

I have little knowledge of entity–attribute–value model databases, and I know that writing direct SQL queries depends on the particular Magento installation and whatnot so I would prefer to use the models.

This import process will be a cron job that runs every 60 minutes.

I know this depends on how my code turns out, but if I'm importing up to 100 products, orders, and customers per run, will using Magento's ORM be extremely resource intensive and slow?

Answer 1

There's always a performance/complexity/fidelity tradeoff with Magento, and the only way to overcome it is to pull up your sleeves, suffer through a few bad imports, and learn what works best for your situations.

Two things that may be of interest to you. First, make sure your WSDL cache is on. Many of the V2 SOAP API performance problems can be traced back to PHP not caching the WSDL file.

Second, considering writing native PHP scripts that use the native PHP API objects. I don't mean models like Mage_Catlog_Model_Product, but API objects like

$model = Mage::getModel('customer/customer_api');
var_dump(get_class($model);

These objects are where the business logic for RPC/SOAP API code lives, and if you can call them directly you avoid a lot of the performance problems related to the WS API overhead.