Mongo Reporting Out of Space on AWS

Question

I'm running Mongo on a small AWS Linux instance. This morning my system shut down with lots of errors in the logs of database locks failing due to disk being full.

How can I figure out why/what is full? I thought small AWS instances were supposed to have 160G available? When I du the mongo /data directory it shows 5.5G used.

I know Mongo can be disk-hungry. My data (raw) is about 150-200MB of JSON.

Answer 1

Your disk is probably full.

The 160 GB number refers to instance storage which is only enabled if you explicitly enable it. The reason why its not enabled by default is that its not persistent. Since this storage is attached to the host, you will loose anything contained in it if you stop your instance or the host fails for any reason.

The storage you are using is an EBS volume. Which is AWS equivalent to a SAN volume. By default, most AMI's create this volume at around 8-10 GB. You can increase its size up to 1TB per volume.

To rectify your current situation, you will need some downtime. You can either do an import into a new instance that is configured with more space. Otherwise, you can snapshot your current volume and use that snapshot to create a new larger volume. You will need to resize the partition to make use of all the space.

Answer 2

As you mentioned, small instance has ephemeral disk locally, but it is mounted to somewhere like /media/ephemeral0, not the root / . By default, the EBS root volume is mounted onto / and the default data directory of MongoDB is /data. That means MongoDB lays in EBS root volume, instead of ephemeral disk.

Besides, ephemeral storage is lost when instances are terminated so it is generally not recommended for use unless you’re comfortable with the data-loss implications.

See also:
MongoDB Document about EC2
AWS Document