Custom storage system for GridFS (MongoDB)?

Question

Can anyone point me to any projects/django apps that provide a plugable custom storage system so I can use GridFS with Django to store file uploads?

I've found django-mongodb but it doesnt seem to support GridFS, nor does django-storages.

I plan to run mysql for the normal database requrements and only use mongodb for file storage so to be clear I dont want to use mongodb as my main database.

Answer 1

I work on PyMongo, the MongoDB Python driver, and haven't heard of any project to provide custom storage for Django using GridFS. This looks like it wouldn't be very hard to write on top of PyMongo: could probably be a direct translation of the GridFS API onto the Django storage API. Maybe could take a look at throwing something together at some point, but this would be a great open-source project for anybody looking to get involved.

Answer 2

I recently implemented the GridFS support in Mongoengine which you might like to checkout. This includes a Django storage backend which you can plug right into you projects and use with ImageField etc. I'm using these technologies in production and it's holding up great so far.

Answer 3

django-mongodb-engine might be worth a look as it enables you to do that without the need to make changes to your existing Django code.

Answer 4

I needed exactly that for Mayan EDMS, plugable storage and database separation. Using Michael Dirolf's latest PyMongo library, it was rather trivial to get a basic class going.

To use it:

from gridfsstorage import GridFSStorage
file = models.FileField(storage=GridFSStorage())

the gridfsstorage.py file:

import os

from django.core.files.storage import Storage
from django.utils.encoding import force_unicode
from django.conf import settings

from pymongo import Connection
from gridfs import GridFS

class GridFSStorage(Storage):
    def __init__(self, *args, **kwargs):
        self.db = Connection(host=settings.GRIDFS_HOST,
            port=settings.GRIDFS_PORT)[settings.DATABASE_NAME]
        self.fs = GridFS(self.db)


    def save(self, name, content):
        while True:
            try:
                # This file has a file path that we can move.
                if hasattr(content, 'temporary_file_path'):
                    self.move(content.temporary_file_path(), name)
                    content.close()
                # This is a normal uploadedfile that we can stream.
                else:
                    # This fun binary flag incantation makes os.open throw an
                    # OSError if the file already exists before we open it.
                    newfile = self.fs.new_file(filename=name)
                    try:
                        for chunk in content.chunks():
                            newfile.write(chunk)
                    finally:
                        newfile.close()
        except Exception, e:
            raise
        else:
            # OK, the file save worked. Break out of the loop.
            break

        return name


    def open(self, name, *args, **kwars):
        return self.fs.get_last_version(name)


    def delete(self, name):
        oid = self.fs.get_last_version(name)._id
        self.fs.delete(oid)


    def exists(self, name):
        return self.fs.exists(filename=name)        


    def path(self, name):
        return force_unicode(name)


    def size(self, name):
        return self.fs.get_last_version(name).length

    def move(self, old_file_name, name, chunk_size=1024*64):
        # first open the old file, so that it won't go away
        old_file = open(old_file_name, 'rb')
        try:
            newfile = self.fs.new_file(filename=name)

            try:
                current_chunk = None
                while current_chunk != '':
                    current_chunk = old_file.read(chunk_size)
                    newfile.write(current_chunk)
            finally:
                newfile.close()
        finally:
            old_file.close()

        try:
            os.remove(old_file_name)
        except OSError, e:
            # Certain operating systems (Cygwin and Windows) 
            # fail when deleting opened files, ignore it.  (For the 
            # systems where this happens, temporary files will be auto-deleted
            # on close anyway.)
            if getattr(e, 'winerror', 0) != 32 and getattr(e, 'errno', 0) != 13:
                raise