Unicrawler and MongoError

Question

I'm using a crawler on my server (FreeBSD 8.3). This crawler (unicrawler https://github.com/MiLk/UniCrawler) is using mongodb but unfortunately I put to much data on the database. And now I have this message :

    /usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/connection/base.js:242
        throw message;      
              ^
MongoError: too much data for sort() with no index.  add an index or specify a smaller limit
    at Object.toError (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/utils.js:110:11)
    at /usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/cursor.js:685:54
    at Cursor.close (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/cursor.js:959:5)
    at commandHandler (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/cursor.js:685:21)
    at /usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/db.js:1806:9
    at Server.Base._callHandler (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/connection/base.js:442:41)
    at /usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/connection/server.js:485:18
    at MongoReply.parseBody (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
    at null.<anonymous> (/usr/UniCrawler/api/node_modules/mongodb/lib/mongodb/connection/server.js:443:20)
    at EventEmitter.emit (events.js:95:17)
npm ERR! weird error 8
npm ERR! not ok code 0

I try to do a make deinstall then make && make install (of node, mongodb, redis and unicrawler) but I still have all the data. How could I wipe all these data ?

Answer 1

If you want to remove the data, you will have to open a shell and delete it.

Assuming mongo is on the path (and the mongo daemon is running), try this on your shell:

$ mongo //This will start a mongo shell
> show dbs; //This will list all the existing dbs

After you found the name of the db you want to delete, Ctrl+D to leave the mongo shell, and type:

$ mongo <dbname> --eval "db.dropDatabase()"

---

Dropping the database will completely remove any trace of data from your system. If, on the other hand, you just want to empty the tables (and keep the dbs), then, after finding the name of your db, type in the mongo shell:

> show tables;
> db.table_name.remove();

Mongo shell reference

Answer 2

Rather than removing all your data, a better option would be to add an appropriate index or limit() to allow your query to accommodate larger result sets.

The "too much data for sort() with no index" error indicates that you are trying to sort a result set larger than 32MB (the limit for an in-memory sort as at MongoDB 2.4). An index will make this query both faster and more scalable.

For further information and examples see the MongoDB Manual: Use Indexes to Sort Query Results.