Greenplum database is not expanding dynamically to our Linux system

Question

The main difficultly in this situation is a out of disk space error from the Greenplum database. We are running a Greenplum Database 4.2.0.0 on a Amazon Linux box.

The investigation has presented the following:

1) Greenplum says we are low on space. We run the gp_disk_free command and get the following results.

0;" sdw1";" /dev/md126";2740872
1;" sdw1";" /dev/md127";1968592
2;" sdw2";" /dev/md126";1100
3;" sdw2";" /dev/md127";1728236

Each line is a segment on the database, sdw1 and sdw2 are the hosts, and the /dev/md* are the devices it says the database is on. The numbers are the number of bytes left in the database. AKA, a few megabytes.

2) We look at our Linux system with df -h and get the following:

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  1.6G  6.3G  21% /
tmpfs                 3.7G     0  3.7G   0% /dev/shm
/dev/xvdo             8.0G   48M  8.0G   1% /home
/dev/md127             24G  456M   24G   2% /data1

Our conclusion is we have enough space for the database to take up, but we cannot figure out how to configure Greenplum to do so dynamically or even how to have it increase in size to take up more filesystem space on the Linux box.

We have investigated the gpexpand command and it seems that only works when you are adding new hosts, and hardware which we do not need to do considering how much space we have on our Linux machine.

We also increased the free space mapping by increasing the max_fsm_pages and max_fsm_relations parameters but that did little to free up space after running a Vacuum command. We are lost with what to do next.

Any ideas on how we can increase the disk size of our Greenplum database on our Linux machine?

Answer 1

gp_disk_free simply runs the df command on each segment. It reports the free size in kilobytes.

You have 4 segments on 2 segment servers. Each segment is using a different disk; that is a good strategy.

Segment 2, which is running on sdw2 and using /dev/md126 is the segment that is low on disk. It has only 1.1 MB.

If I assume that nothing other than the database is using disk space, then there is a problem with skew. Some of your tables have distribution policies that are causing many more rows to be placed on segment 2 than on the other segments.

You can investigate this with something like this:

select gp_segment_id, count(1) from <your_table_name> group by 1 order by 1 ;

The solution is to change your distribution policy. Either choose a different column to place in the "distributed by" clause or use the "distributed randomly" clause.

If it isn't a skew problem, then look at what is consuming disk space on sdw2.

Leonard Walstad

Pivotal

www.gopivotal.com