MongoDB uses the operating system's mmapped IO as storage engine. The theoretical maximum, if there are no other limits imposed by implementation details, would be the amount of virtual memory your OS can assign to the mongod process. On a 32 bit OS, you will run into this limit very quickly. On windows for example, by default half the address space is reserved for the OS, so your mongod instance will stop accepting writes at a size of around 2 Gb
On a 64 bit OS, this theoretical maximum is quite a bit larger - 64 bits worth of it, in the order of millions of terabytes if I'm not mistaken. You will reach another limit before you reach this one.
EDIT:
This may no longer be true as of MongoDB 3.0. The WiredTiger storage engine works in a different way.