Here's a comprehensive guide from their documentation:
https://github.com/mathjax/MathJax-docs/wiki/Guide%3A-reducing-size-of-a-mathjax-installation
They end up with a 1.7MB library size, 700KB after GZIP compression. Of course the amount you'll be able to trim out depends on your specific project requirements.