Broken Pipe when I try to insert large blob with pymysql

Question

I have a flask-based web app that allows users to upload files. Files are stored in a mysql database.

This works fine until the file is larger than around 16Mb, the insert fails with the following:

File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1518, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1506, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1504, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1264, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1262, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1248, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/<redacted>/access_control.py", line 15, in decorated_function
    return f(*args, **kwargs)
  File "/<redacted>/views/files.py", line 48, in upload
    VALUES (%s, %s, %s, %s, %s)""", file_details)
  File "/<redacted>/database.py", line 66, in query
    cursor.execute(sql, values)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 262, in execute
    result = super(DictCursor, self).execute(query, args)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 117, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 187, in defaulterrorhandler
    raise Error(errorclass, errorvalue)
Error: (<class 'socket.error'>, error(32, 'Broken pipe'))

I got excited when the filesize appeared to match the mySQL setting max_allowed_packet, so I changed it in my.cnf and restarted but it didn't help. (show variables like 'max_allowed_packet' shows the new value of 150M)

The file definitely gets uploaded to the server, I put some code into my insert method to write the file to disk before it ran the query and the file was ok.

The field the blob is being inserted into is a longblob, the code responsible for the insert is:

@mod.route('/file/upload', methods=['POST'])
@login_required
def upload():
    filename = request.files['file'].filename
    mime_type = request.files['file'].mimetype
    #filesize = request.files['file'].content_length
    file = request.files['file'].stream.read()

    if mime_type[:5] == 'image':
        file = resize_image_to_width(file, 1024)

    filesize = len(file)
    if filesize == 0:
        return ""

    if not request.form['file_id']:
        file_details = (filename, file, mime_type, filesize, session['user']['user_id'])
        file_id = database.query("""INSERT INTO files (filename, file, mime_type, filesize, owner)
                                      VALUES (%s, %s, %s, %s, %s)""", file_details)
    else:
        file_details = (filename, file, mime_type, filesize, request.form['file_id'])

        file_id = database.query("""UPDATE files 
                                    SET 
                                        filename=%s, 
                                        file=%s, 
                                        mime_type=%s, 
                                        filesize=%s 
                                    WHERE file_id=%s""", file_details)
    return "upload complete"

I'm at a bit of a loss now, I did find some stuff a week or so ago that suggested that the file needed to be inserted in sections, but I can't find it now (I got distracted by my actual job!) and I don't know how to go about inserting it in chunks.

I would really appreciate some help with this!

Answer 1

First you want to chunk your file into blocks that are a safe size to insert, in your case you could make that 10MB. You can use the chunks function in that answer to do this. Then you insert your first chunk of data and do a bunch of updates that concat the additional chunks onto that blob. To do that portion check out the question on append/concatenate BLOB data to a BLOB column using update?.

Answer 2

This is another approach. Load the file into the blob column using the LOAD_FILE function.

UPDATE t SET blob_col=LOAD_FILE('/tmp/image.png') WHERE id=1;

This shouldn't face the 16MB limitation you had. If you face any binary encoding issues you can save the image with python in hex format and then use UNHEX to safely decode it back to it's raw binary form.

UPDATE t SET blob_col=UNHEX(LOAD_FILE('/tmp/image.png.hex')) WHERE id=1;

Both options though need you to save the file on your MySQL server filesystem if your webserver is on the same server that this is trivial otherwise you might have to use something like scp or ftp to transfer the file to the MySQL server before issuing the SQL statement.