As you've posted here, the compression function is called "squash", a function part of AMOS Pro.
As such, my advice would be to try one of the following lines of attack:
Reverse engineer the algorithm by analyzing its output: This is definitely not a viable option. You will only waste time.- Read, annotate, understand the source code of the unsquash function in AMOS Pro
- Contact the author of AMOS Pro
Read the source code
The source code for AMOS Pro is apparently in the public domain now and can be found here:
http://www.pianetaamiga.it/downloads/AMOSPro_Sources.zip
It consists of 68000 assembly code and quite a few compiled object files.
The unsquash function can be found in the file +header.s
on line 1061 and onwards. It is not documented, except for its entry register values, which is good at least. It doesn't appear to be a very large function so this might be worth a shot.
You will need to have, or obtain/learn, rudimentary 68000 machine code. It does not appear to call out to system libraries or anything and only seem to operate directly on memory, which would suggest this is actually doable (ie. understanding the code). Still, I've never written or read 68000 code in my life so what do I know.
Contact the author of AMOS Pro
The author of AMOS Pro is François Lionet, as is evident by the User Guide, he founded Clickteam in the mid-90s to make game- and multimedia-making software. He still seems to be situated in that company and according to forum posts from others looking into AMOS Pro he seems to be willing to answer email. Sadly I don't know his email but the Clickteam website above should give you a starting point.