How to overstep maxsize in python?

Question

It is said from http://docs.python.org/2/library/sys.html that maxsize is [...] the maximum size [...] ! And guess what ? I need to overstep it !

On my 64 bits system, it comes that maxsize = 2**63-1 = 9223372036854775807

If an array is maxsized, is a N-dimensional array N-maxsized ?

Because in python an N-dimensionnal array is an array into an array, I guess that code is valuable in order to overstep the maxsize limit ?

Understoud that I need memory ...

array=[]
i,j=0
while j<(2**63-1):
   while i<(2**63-1):
      array[j].append(1)
      i=i+1
j=j+1

Answer 1

Bugs and legacy cruft non withstanding, maxsize is (approximately) the largest size a container¹ can be. You cannot have a list that large even on a 64 bit system, for simple reasons of address space limitations (and some choices in the implementation, but those are a consequence of the aforementioned limit). You can't even store 2^63-1 bytes on your computer (not even on the hard disk!), much less 2^63-1 fully blown Python objects.

You have no idea how much memory your example code needs that. Your CPU can't even address a tiny fraction of the memory needed for that. Not even your whole hard disk would be able to hold a significant fraction. You'd need 8 million hard disks of 1 TiB capacity each to store 2^63-1 bytes, and 8 times that for equally many 1s. That exceeds even the alleged capacity of the NSA's Utah data center by far. And that's 2^63-1, not 2^63 * 2^63 = 2^126. The latter figure is ridiculous to think of in physical terms. This is in the order of magnitude that cannot be enumerated or brute forced and can be randomly sampled without getting duplicates.

If that physical limit didn't exist, or if an artificial limit on list size existed that was smaller than the actual physical limit, you could indeed use a list within a list to sidestep the problem. This is unlikely to be an actual problem in practice though. The minimum limit on container size is around 2 billion.

¹ Under the assumption that each element of a container physically exists in RAM, not moved to the disk or calculated on demand or anything.