Error Mongo: allowDiskUse:true to opt in

[rexdouglass]

~/duplicate-images$ python3 duplicate_finder.py find
Started database...
Traceback (most recent call last):
File "duplicate_finder.py", line 358, in
dups = find(db, args['--match-time'])
File "duplicate_finder.py", line 238, in find
"total": {"$gt": 1}
File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 2397, in aggregate
**kwargs)
File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 2304, in _aggregate
client=self.__database.client)
File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged)
File "/usr/local/lib/python3.5/dist-packages/pymongo/network.py", line 150, in command
parse_write_concern_error=parse_write_concern_error)
File "/usr/local/lib/python3.5/dist-packages/pymongo/helpers.py", line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.

[senmaoy]

I have the same problem when processing a 50G dataset!

[7633]

I've just add allowDiskUse=True to db.aggregate method:

def find(db, match_time=False):
    dups = db.aggregate([{
        "$group": {
            "_id": "$hash",
            "total": {"$sum": 1},
            "items": {
                "$push": {
                    "file_name": "$_id",
                    "file_size": "$file_size",
                    "image_size": "$image_size",
                    "capture_time": "$capture_time"
                }
            }
        }
    },
    {
        "$match": {
            "total": {"$gt": 1}
        }
    }], allowDiskUse=True)

    if match_time:
        dups = (d for d in dups if same_time(d))

    return list(dups)