Create database table from CSV

Question

I am working on a module that requires access to some tabular data provided from an external source. It's only two columns but has roughly 40000 rows.

Currently, my module is just parsing the CSV in entirety when necessary. This works well, but since the file is about 450Kb, this will eat up server resources once deployed to a production site.

I'd like to move this data to a Magento table and am having trouble.

I'm using the RDBMS methods in my setup script, ala:

$installer = $this;
$installer->startSetup();

$table = $installer->getConnection()
    ->newTable($installer->getTable('my_table'))
    ->addColumn('column_a', Varien_Db_Ddl_Table::TYPE_TEXT, 5, array(
        'nullable' => false,
        ), 'Column A')
    ->addColumn('column_b', Varien_Db_Ddl_Table::TYPE_TEXT, 3, array(
        'nullable' => false,
        ), 'Column B');

$installer->getConnection()->createTable($table);

$installer->endSetup();

This works just fine to create my table, but actually getting my data in is wracking my head (I'm still a noob when it comes to database management).

Ideally, I would just parse my CSV file and insert values into the table without having to deal with a bunch of copy/paste.

Are there any built-in methods for handling arbitrary CSV data, or are they all built to handle just what they need to? What would be the best approach to get my data into Magento?

Answer 1

Never done this. Let's play!

After the createTable() call, or in a subsequent script:

/* @var $installer Mage_Core_Model_Resource_Setup */
$installer = $this;
$installer->startSetup();

//Extract data from CSV file
$csv = new Varien_File_Csv;
$data = $csv->getData('path/to/file.csv');

$resultNum = $installer->getConnection()->insertArray(
    $installer->getTable({TABLE NAME}),
    array({COL 1},{COL 2}),    //column names
    $data
);

//EDIT: if this is done using a data setup script, can even log the results:
Mage::log(
    __FILE__." added $resultNum records to $installer->getTable({TABLE NAME})",
    Zend_Log::INFO,
    "setup.log",
    true
);

$installer->endSetup();