I would flip this to two operations to make this atomic: 1) find 2) if not found then UPSERT
EDIT: Or, use a UNIQUE index and catch/dismiss unique exceptions thrown.
EDIT2:
To make this a "real answer" I'll elaborate.
To keep this a one transaction operation, you can ensure a unique index (http://docs.mongodb.org/manual/tutorial/create-a-unique-index/) on the fields you wish. Using find_or_create_by in a multithreaded fashion will work correctly, as long as you have the ability to catch and dismiss exceptions/errors that occur when a duplicate key is found (you'll have to look into the language your using)
Or convert this to two transactions. First do a find based on your criteria, if no result is found do an insert. Doing a normal insert here would also produce issues, however using an upsert operation (update with upsert option - http://docs.mongodb.org/manual/reference/method/db.collection.update/) this will insure uniqueness.