Is it possible to prevent data loss with Postgres WAL archiving?

Question

As I understand from the docs, a WAL segment is archived when the segment file reaches some size or due to archive_timeout.

So if the database crashes, all the data from the incomplete, currently active WAL segment would be lost (assuming we don't have any access to that incomplete segment in the Postgres directory).

Do I get it right and if so, is there a way to avoid this data loss?

Answer 1

You are correct.

The way to avoid this is to use pg_receivewal, which streams WAL information from the database server and persists it in WAL segment copies immediately.

By default this is asynchronous, so you could lose some split seconds in case of a crash. It you need to avoid that, you'd have to use synchronous replication with pg_receivewal, but be warned that

1. This makes DML statements slower, since COMMIT cannot succeed until it has heard back from pg_receivewal.

2. you shouldn't set synchronous_commit = remote_apply, because pg_receivewal never applies the changes, it just saves them to a WAL file.

Answer 2

PostgreSQL uses Write-Ahead Logging (WAL) as its approach to transaction logging. WAL's central concept is that changes to data files (where tables and indexes reside) must be written only after those changes have been logged, that is, when log records describing the changes have been flushed to permanent storage. If we follow this procedure, we do not need to flush data pages to disk on every transaction commit, because we know that in the event of a crash we will be able to recover the database using the log: any changes that have not been applied to the data pages can be redone from the log records. (This is roll-forward recovery, also known as REDO.) This is from link below:

https://wiki.postgresql.org/wiki/PostgreSQL_for_Oracle_DBAs

But anything coming from memory to disk will start at redo plus databases only guarantee committed transactions. Anything which is only in memory is not committed and non-recoverable. Some HA mechanisms do exist for saving transaction in memory but not PostgreSQL.