Initializing data structures from large static lists and maps in Haskell

Question

I am translating the zxcvbn password strength algorithm to Haskell.

In the first step, I generate several large data structures from common word lists (English words, surnames, male names, etc). These lists are static and will only be rebuilt occasionally, and then only when the application is recompiled.

These Lists and Maps are use to initialize the actual password strength algorithm.

When I converted the original JavaScript to Scala, I saved the static values as JSON that gets loaded by the algorithm when it is launched. I wanted to just generate initialized Scala vals, but the Java Virtual Machine has a 64KByte limit on classes, and my data exceeded that (each structure contains thousands of words and is bigger than 64KBytes).

What is the best way to handle this in Haskell? Can I just generate a source code module containing the data? What is the maximum module size limit? I searched the Haskell2010 report, but could not find a compiler limits section.

Generating Haskell source code would obviously be my first choice, as then the code can remain pure. If I need to load JSON and parse it on start-up, that part of the the module will be impure.

Answer 1

Can I just generate a source code module containing the data?

Yes. There's no limits -- though your specific machine may have trouble with some types of data and some optimizatino levels.

Bytestring literals of the serialized data generally "just work".

See the earlier question - How to compile a resource into a binary in Haskell? and on the Haskell wiki