Django with Pluggable MongoDB Storage troubles

Question

I'm trying to use django, and mongoengine to provide the storage backend only with GridFS. I still have a MySQL database.

I'm running into a strange (to me) error when I'm deleting from the django admin and am wondering if I am doing something incorrectly.

my code looks like this:

# settings.py
from mongoengine import connect
connect("mongo_storage")

# models.py
from mongoengine.django.storage import GridFSStorage
class MyFile(models.Model):
    name = models.CharField(max_length=50)
    content = models.FileField(upload_to="appsfiles", storage=GridFSStorage())
    creation_time = models.DateTimeField(auto_now_add=True)
    last_update_time = models.DateTimeField(auto_now=True)

I am able to upload files just fine, but when I delete them, something seems to break and the mongo database seems to get in an unworkable state until I manually delete all FileDocument.objects. When this happens I can't upload files or delete them from the django interface.

From the stack trace I have:

/home/projects/vector/src/mongoengine/django/storage.py in _get_doc_with_name
        doc = [d for d in docs if getattr(d, self.field).name == name] ...
▼ Local vars
Variable    Value
_[1]    
[]
d   

docs    
Error in formatting: cannot set options after executing query
name    
u'testfile.pdf'
self    

/home/projects/vector/src/mongoengine/fields.py in __getattr__
        raise AttributeError 

Am I using this feature incorrectly?

UPDATE:

thanks to @zeekay's answer I was able to get a working gridfs storage plugin to work. I ended up not using mongoengine at all. I put my adapted solution on github. There is a clear sample project showing how to use it. I also uploaded the project to pypi.

Another Update:

I'd highly recommend the django-storages project. It has lots of storage backed options and is used by many more people than my original proposed solution.

Answer 1

I think you are better off not using MongoEngine for this, I haven't had much luck with it either. Here is a drop-in replacement for mongoengine.django.storage.GridFSStorage, which works with the admin.

from django.core.files.storage import Storage
from django.conf import settings

from pymongo import Connection
from gridfs import GridFS

class GridFSStorage(Storage):
    def __init__(self, host='localhost', port=27017, collection='fs'):
        for s in ('host', 'port', 'collection'):
            name = 'GRIDFS_' + s.upper()
            if hasattr(settings, name):
                setattr(self, s, getattr(settings, name))
        for s, v in zip(('host', 'port', 'collection'), (host, port, collection)):
            if v:
                setattr(self, s, v)
        self.db = Connection(host=self.host, port=self.port)[self.collection]
        self.fs = GridFS(self.db)

    def _save(self, name, content):
        self.fs.put(content, filename=name)
        return name

    def _open(self, name, *args, **kwars):
        return self.fs.get_last_version(filename=name)

    def delete(self, name):
        oid = fs.get_last_version(filename=name)._id
        self.fs.delete(oid)

    def exists(self, name):
        return self.fs.exists({'filename': name})

    def size(self, name):
        return self.fs.get_last_version(filename=name).length

GRIDFS_HOST, GRIDFS_PORT and GRIDFS_COLLECTION can be defined in your settings or passed as host, port, collection keyword arguments to GridFSStorage in your model's FileField.

I referred to Django's custom storage documenation, and loosely followed this answer to a similar question.