Running moses server on Amazon

Question

I am trying to run a moses server on Amazon ec2 ebs-backed instance. The languages models and translation models are about 200GB in total. I am thinking to have a moses installed instance loads languages models and translation models stored on s3. But i do not know how to configure moses.ini file in order to make moses knowing the path of ttable-file and lmodel-file. If anyone has done this before, any help would be greatly appreciated!!

Thank you.

Answer 1

I wouldn't recommend Amazon S3 for this. Amazon S3 is used for efficiently distributing files across the web. But if your whole purpose is to just read these files inside a VM - then saving these in S3 is not the correct choice. Refer to this answer for more details.

To answer your question, yes it is possible to mount an S3 bucket as a folder inside your server using S3FS. Here are instructions for Ubuntu and Red Hat.

But other ideal approaches are:

• If you don't have enough space in the hard disk, then install Moses server on a different partition, format it using BTRFS and enable Transparent Compression. This will automatically compress/decompress files when you save/retrieve from the hard disk, so you will end up using much much less space. Also in a lot of benchmarks, Transparent Compression is shown to be faster, since lesser amount of data is transferred between hard disk and RAM. Specially when involving large files.

• You can always attach a secondary EBS disk to your running VM (like a secondary hard disk). Use that for storing the translations/models (and you can also combine enable transparent compression as above too)

• Run a separate VM without EBS, and just using the normal instance storage, and use that to store the translations alone. Now in your Moses server, you can mount the translations alone from this separate non-EBS VM using SSHFS

Overall, don't use S3, there are other much better ways.

Edit: Added link