Squeak literal limit

Question

I have now run for the third time into this problem some of you Squeak-programmers might recognize best from a screenshot:

Here is example-code to reproduce the problem:

d:=Dictionary new.
(1 to: 257) do: [:each|
    d at: each put: each
].
f:=FileStream fileNamed: 'asdf.txt'.
d storeOn: f.
f reset.
d2:=Dictionary readFrom: f.
f close.

I know why it occurs (the compiler is used to de-serialize and it does so by putting the whole file in a method).

In the past I always changed my code to either use a DB or use some other form of serialization. But i would like to know if there is some way to fix Squeak to make this work instead.

Answer 1

Currently, there is no proper way around that.

The problem is, that CompiledMethods have a format that is known by the VM, and also expected by the VM. That includes the 256 literals limit. A change to the format of methods requires changes to the virtual machine. That is why people have hesitated to change the literal limit up until now.

Recently, people have spent energy to create a new format not only for methods but to change the Object format altogether: the SPUR object format is intended to increase the number of literals in methods, among other things.

Answer 2

As illustrated, the #storeOn: APIs aren't hardened serialization APIs. But if you're going to use them (instead one of the serialization projects that comes and goes every year or so), you can fight "clever" (which is what the 256 literal limit is, it's an optimization thing), with some "clever" of your own.

One trick, is to leverage that there is not a limit on what we might think of as "nested" literals.

a := #(1 2 3 4 5 6 7 8 9)

does not contain 10 literals. It just contains one, the array itself. It can have as many literals inside of it as it wants. So leveraging a little extra Dictionary mojo, we can change your example to be:

d := Dictionary new.
(1 to: 257) do: [:each | d at: each put: each].
f := FileStream fileNamed: 'asdf.txt'.
pairs := d associations collect: [:kv | Array with: kv key with: kv value ].
pairs storeOn: f.
f reset.
d2 := Dictionary newFromPairs: (Array readFrom: f).
f close.
d2

Basically we're turning your dictionary into an Array of Arrays. So it's just one literal that you're going to store. And then we're using the handy newFromPairs: which just happens to take ... drum roll ... an Array of Arrays.