Problems running AUTO_VACUUM (to prevent wrap-around) on a very large table

Question

I'm having a problem and need some advice. I am normally a developer, however with recent staff changes at my company I am now the sole IT person, so I'm having to branch into a lot of unknown areas and really need some help.

We are running postgres 8.3. The database is trying to run AUTO_VACUUM on a large object table (pg_catalog.pg_large_object) to prevent transaction ID wrap-around. I think I understand the basics of what that means. The problem is, this table is 750G with 452 million rows. AUTO_VACUUM is writing to disk a lot, and eating up disk space (it consumed our last 250GB of 1TB yesterday). After an emergency outage, we are back up and running with 1100GB of space, and 100GB free. However, once postgres was back up and running, it kicked off the AUTO_VACUUM process again. If I kill the transaction (which I'm sure is not recommended), it just restarts.

So here are my questions:

1) For that table, how much space would it need to finish the AUTO_VACUUM process? How do I determine this?

2) Is there a better way to configure the server to handle this situation so it doesn't require ridiculous amounts of disk space whenever it needs to do this?

3) If no to 2, how do you propose this problem get fixed?

I am not a DBA, and do not have linux server administration experience, just a developer being asked to wear many hats. I'm trying to get a DBA consultant to help resolve the problem, but the company is pushing back. They don't seem to understand the severity of the problem, despite my best efforts.

Suggestions? Comments? Any advice or guidance you can provide would be greatly appreciated. If you need more info, let me know.

Answer 1

If you do not resolve this problem fairly quickly your database will go into emergency shutdown to prevent data corruption and refuse to start back up until txid wraparound vaccuum completes. Check the logs to see how close to this point you are, you'll see messages like:

WARNING:  database "mydb" must be vacuumed within 177009986 transactions 
HINT:  To avoid a database shutdown, execute a database-wide VACUUM in "mydb". 

Do not just kill the vacuum and put the problem off. You really, really need to resolve this unless you can afford some unplanned downtime.

The reason it's consuming tons of disk space is probably that you're on an old version that doesn't have auto-managed freespacemap settings, and you've likely exceeded max_fsm_pages and/or max_fsm_relations. Check the log, you may well see messages about these.

Unfortunately you can't just raise these params after the fact. This old PostgreSQL install has lost the knowledge about what space in the table is free. Proper cleanup and recovery will require a CLUSTER of the table, which needs at least as much free space as the table+index sizes and requires an exclusive lock on the table for the duration of the run.

Most of the less intrusive mitigating options like pg_reorg are no longer open to you now that you're approaching forced txid wraparound prevention. Your best bet is quite probably to give autovacuum the space it needs to complete the job - or to deal with the downtime and CLUSTER then VACUUM FREEZE the table to get the process over and done with faster.

Once you've recovered, I would recommend greatly increasing max_fsm_pages and making sure max_fsm_relations is big enough. Lots of tuning advice for these old versions is around, search.

Plan an upgrade to 9.2, which auto-manages the freespace map (as does any version 8.4+) and has all sorts of autovac enhancements to help stop you getting into these pickles in the first place.

If this situation is desperate consider getting in touch with professional PostgreSQL support provider. (Proper disclosure: I work for 2ndQuadrant, one of the listed providers).

Answer 2

The real-time support on FreeNode's #postgresql (IRC) is amazing. There are frequently knowledgable people awake and available to talk DBA/development details. I can't recommend it enough.