ARPA language model documentation

Question

Where can I find documentation on ARPA language model format?

I am developing simple speech recognition app with pocket-sphinx STT engine. ARPA is recommended there for performance reasons. I want to understand how much can I do to adjust my language model for my custom needs.

All I found is some very brief ARPA format descriptions:

• http://kered.org/blog/2008-08-12/arpa-language-model-file-format/
• http://www.speech.sri.com/projects/srilm/manpages/ngram-format.5.html
• http://www.speech.cs.cmu.edu/SLM/toolkit_documentation.html

I am beginner to STT and I have trouble to wrap head around this (n-grams, etc...). I am looking for more detailed docs. Something like documentation on JSGF grammar here:

http://www.w3.org/TR/jsgf/

Answer 1

There is actually not much more to say about the format than is said in those docs..

Besides, you'll probably want to prepare a text file with sample sentences and generate the language file based on it. There is an online version which can do it for you: lmtool

Answer 2

You can complement those docs with this tech report that gives a comprehensive overview of smoothing for language modeling: http://www.ee.columbia.edu/~stanchen/papers/h015a-techreport.pdf You will also find definitions for backoff models and interpolated models.

Answer 3

I'm probably very late to answer this, I found the ARPA LM format well documented in this link from The HTK Book by Steve Young et. al.

Each line of ARPA is a triple that stores:

n-gram log-probability(base10) ; the n-gram itself ; back-off weight (also in log space).