Python2.7 MySQL Connector Error on LOAD DATA LOCAL INFILE

Question

I'm trying to dynamically load census data into a mysql database (from .csv files) using Python and MySQL Connector.

I can't figure out why I am getting the error:

Traceback (most recent call last):
  File "miner.py", line 125, in <module>
    cursor.execute(add_csv_file, csv_info)
  File "/Library/Python/2.7/site-packages/mysql/connector/cursor.py", line 393, in execute
    self._handle_result(self._connection.cmd_query(stmt))
  File "/Library/Python/2.7/site-packages/mysql/connector/connection.py", line 586, in cmd_query
statement))
  File "/Library/Python/2.7/site-packages/mysql/connector/connection.py", line 508, in _handle_result
    raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '/Users/afink/Documents/Research/Current Papers & Books/Youth Assessm' at line 1

The outputted string before execution works fine in MySQL command line interface under the same user.

Seems like it should be a simple problem, but I'm stuck!

def db_connect():
    config = {
        'user': 'username',
        'password': 'pass',
        'host': 'localhost',
        'database': 'uscensus',
        'raise_on_warnings': True,
    }

    from mysql.connector import errorcode
    try:
      cnx = mysql.connector.connect(**config)
      return cnx
    except mysql.connector.Error as err:
      if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
        print("Something is wrong with your user name or password")
      elif err.errno == errorcode.ER_BAD_DB_ERROR:
        print("Database does not exist")
      else:
        print(err)
    else:
      cnx.close()


cursor = cnx.cursor()

os.chdir("./")
for files in glob.glob("*.csv"):

    add_csv_file = ('''load data local infile '%(cwd)s/%(file_name)s' into table %(table_name)s 
                    columns terminated by ',' 
                    optionally enclosed by '\"'
                    escaped by '\"'
                    lines terminated by '\\n';''')

    # Insert CSV file information
    csv_info = {
        'cwd': os.getcwd(),
        'file_name': files,
        'table_name': 'SF1_' + files[2:-8],
    }


    print add_csv_file % csv_info # Temporary Debug

    cursor.execute(add_csv_file, csv_info)

# Make sure data is committed to the database
cnx.commit()
cursor.close()
cnx.close()

Thank you for your help in advance!

Answer 1

This is easily solved with adding the appropriate client flag in your connection as below:

import mysql.connector
from mysql.connector.constants import ClientFlag

cnx = mysql.connector.connect(user='[username]', password='[pass]', host='[host]', client_flags=[ClientFlag.LOCAL_FILES])
cursor = cnx.cursor()

This will give permission for MySQL to access the local files on your machine and then the following LOAD will work:

LoadSQL = """LOAD DATA LOCAL INFILE '%s'
    INTO TABLE %s
    FIELDS TERMINATED BY '\t'
    LINES TERMINATED BY '\n'
    IGNORE 1 LINES
    (field1, field2, field3, field4)""" % (csvfile, tabl_name)
cursor.execute(LoadSQL)
cnx.commit()
cursor.close()
cnx.close()

Answer 2

Add field allow_local_infile = "True" when you do mysql.connector.connect. It'll work

Answer 3

Okay, here's what I've figured out.

@Dd_tch - your answer helped me realize that this code would work better:

query = add_csv_file % csv_info
cursor.execute(query)

Though the MySQL site for Connector seems to indicate you could do what I was doing (http://dev.mysql.com/doc/connector-python/en/connector-python-example-cursor-transaction.html), that wasn't working.

When I fixed that, I got a new error: mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version

There are several sites that indicate that this could be fixed by adding "local_infile=1" to the MySQL Connector config. Here's one: http://dev.mysql.com/doc/refman/5.1/en/loading-tables.html

This solution did not work for me. My local_infile is set to 1 on MySQL and I can't set it within the Python code or I get an e

I'm going to instead replace the LOAD LOCAL DATA INFILE with something that will read CSV files line by line and insert lines into the database. This will make the code more portable to other DBs anyway.

Answer 4

You have an error when calling the execute() method:

cursor.execute(add_csv_file, csv_info)

try to:

cursor.execute(query, (add_csv_file, csv_info,))