Raspbian is actually Debian. If you install the same version (7 or 8) on your host, then you can get a toolchain via Emdebian. See this wiki for more details. Example:
apt-get install g++-4.4-arm-linux-gnueabi
Via xapt
you install missing dependencies for cross-compilation.
Alternative approach were to use Buildroot. On one hand BR also has ct-ng backend, on the other hand you can create a complete root file system with it. BR also provides ffmpeg package.
.