How to upgrade postgres 9.5 with non default data directory and checksum enabled to 10.1

https://dba.stackexchange.com/questions/190901

10-10-2020
|

Question

I am trying to upgrade postgres 9.5.4 to 10.1 on Ubuntu 14. The data directory for 9.5.4 is /var/lib/postgresql/data/postgresql0

The steps that I have followed

Install postgres 10.1

ubuntu@van-platform:~/build-target/launcher$sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
ubuntu@van-platform:~/build-target/launcher$wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
ubuntu@van-platform:~/build-target/launcher$ OK
ubuntu@van-platform:~/build-target/launcher$sudo apt-get update
...
ubuntu@van-platform:~/build-target/launcher$sudo apt-get install postgresql-10
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  comerr-dev krb5-multidev libgssrpc4 libkadm5clnt-mit9 libkadm5srv-mit9
  libkdb5-7
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libpq-dev libpq5 postgresql-client-10 postgresql-client-common
  postgresql-common
Suggested packages:
  postgresql-doc-10 locales-all
The following NEW packages will be installed:
  postgresql-10 postgresql-client-10
The following packages will be upgraded:
  libpq-dev libpq5 postgresql-client-common postgresql-common
4 upgraded, 2 newly installed, 0 to remove and 14 not upgraded.
Need to get 6,595 kB of archives.
After this operation, 26.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main libpq-dev amd64 10.1-1.pgdg14.04+1 [241 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main libpq5 amd64 10.1-1.pgdg14.04+1 [157 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main postgresql-common all 188.pgdg14.04+1 [220 kB]
Get:4 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main postgresql-client-common all 188.pgdg14.04+1 [81.5 kB]
Get:5 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main postgresql-client-10 amd64 10.1-1.pgdg14.04+1 [1,282 kB]
Get:6 http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg/main postgresql-10 amd64 10.1-1.pgdg14.04+1 [4,613 kB]
Fetched 6,595 kB in 4s (1,473 kB/s)        
Preconfiguring packages ...
(Reading database ... 102799 files and directories currently installed.)
Preparing to unpack .../libpq-dev_10.1-1.pgdg14.04+1_amd64.deb ...
Unpacking libpq-dev (10.1-1.pgdg14.04+1) over (9.5.4-1.pgdg14.04+1) ...
Preparing to unpack .../libpq5_10.1-1.pgdg14.04+1_amd64.deb ...
Unpacking libpq5:amd64 (10.1-1.pgdg14.04+1) over (9.5.4-1.pgdg14.04+1) ...
Preparing to unpack .../postgresql-common_188.pgdg14.04+1_all.deb ...
Leaving 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (188.pgdg14.04+1) over (175.pgdg14.04+1) ...
Preparing to unpack .../postgresql-client-common_188.pgdg14.04+1_all.deb ...
Unpacking postgresql-client-common (188.pgdg14.04+1) over (175.pgdg14.04+1) ...
Selecting previously unselected package postgresql-client-10.
Preparing to unpack .../postgresql-client-10_10.1-1.pgdg14.04+1_amd64.deb ...
Unpacking postgresql-client-10 (10.1-1.pgdg14.04+1) ...
Selecting previously unselected package postgresql-10.
Preparing to unpack .../postgresql-10_10.1-1.pgdg14.04+1_amd64.deb ...
Unpacking postgresql-10 (10.1-1.pgdg14.04+1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libpq5:amd64 (10.1-1.pgdg14.04+1) ...
Setting up libpq-dev (10.1-1.pgdg14.04+1) ...
Setting up postgresql-client-common (188.pgdg14.04+1) ...
Installing new version of config file /etc/postgresql-common/user_clusters ...
Setting up postgresql-common (188.pgdg14.04+1) ...

Configuration file '/etc/logrotate.d/postgresql-common'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** postgresql-common (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/logrotate.d/postgresql-common ...
Replacing config file /etc/postgresql-common/createcluster.conf with new version
 * Starting PostgreSQL 9.5 database server                               [ OK ] 
Setting up postgresql-client-10 (10.1-1.pgdg14.04+1) ...
Setting up postgresql-10 (10.1-1.pgdg14.04+1) ...
Creating new PostgreSQL cluster 10/main ...
/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/10/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/10/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start

Ver Cluster Port Status Owner    Data directory              Log file
10  main    5433 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
 * Starting PostgreSQL 10 database server                                [ OK ] 
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...

Create the new data directory

support@vrni-platform:~$ sudo -H -u postgres mkdir -p /var/lib/postgresql/10/data/postgresql0

Stop postgres

ubuntu@vrni-platform:~/build-target/launcher$ sudo service postgresql stop
     * Stopping PostgreSQL 10 database server                                [ OK ] 
     * Stopping PostgreSQL 9.5 database server                               [ OK ]

Enable Checksum

support@vrni-platform:/tmp$ sudo -H -u postgres /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/data/postgresql0 initdb -o '--data-checksums'
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /var/lib/postgresql/10/data/postgresql0 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/data/postgresql0 -l logfile start

On trying to check for upgrade viability with the below command

sudo -H -u postgres /usr/lib/postgresql/10/bin/pg_upgrade \
   -b /usr/lib/postgresql/9.5/bin \
   -B /usr/lib/postgresql/10/bin \
   -d /var/lib/postgresql/data/postgresql0 \
   -D /var/lib/postgresql/10/data/postgresql0 \
   -o ' -c config_file=/etc/postgresql/9.5/main/postgresql.conf' \
   -O ' -c config_file=/etc/postgresql/10/main/postgresql.conf' --check

It is failing with the below error in pg_upgrade_server.log

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -b  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/tmp'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2017-11-14 15:27:31 UTC [16239-1] LOG:  database system was shut down at 2017-11-14 15:16:43 UTC
2017-11-14 15:27:31 UTC [16239-2] LOG:  MultiXact member wraparound protections are now enabled
2017-11-14 15:27:31 UTC [16238-1] LOG:  database system is ready to accept connections
........................................................... stopped waiting
pg_ctl: could not start server

Can someone let me know what I am doing wrong and how can I get around this error?

EDIT

I have made the changes to data_directory as suggested in an answer. But now I am getting the below error

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
connection to database failed: could not translate host name "." to address: No address associated with hostname
Failure, exiting

I have made listen_addresses='*' in both the postgres.conf. But still getting the same issue.

Below is the content of pg_upgrade_server.log

-----------------------------------------------------------------
  pg_upgrade run on Tue Nov 14 15:22:47 2017
-----------------------------------------------------------------


-----------------------------------------------------------------
  pg_upgrade run on Tue Nov 14 15:27:30 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -b  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/tmp'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2017-11-14 15:27:31 UTC [16239-1] LOG:  database system was shut down at 2017-11-14 15:16:43 UTC
2017-11-14 15:27:31 UTC [16239-2] LOG:  MultiXact member wraparound protections are now enabled
2017-11-14 15:27:31 UTC [16238-1] LOG:  database system is ready to accept connections
........................................................... stopped waiting
pg_ctl: could not start server
Examine the log output.



-----------------------------------------------------------------
  pg_upgrade run on Tue Nov 14 15:31:35 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -b  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/tmp'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2017-11-14 15:31:35 UTC [21577-1] FATAL:  lock file "postmaster.pid" already exists
2017-11-14 15:31:35 UTC [21577-2] HINT:  Is another postmaster (PID 16238) running in data directory "/var/lib/postgresql/9.5/main"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.



-----------------------------------------------------------------
  pg_upgrade run on Tue Nov 14 15:35:01 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -b -cconfig_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/tmp'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2017-11-14 15:35:01 UTC [25230-1] FATAL:  lock file "postmaster.pid" already exists
2017-11-14 15:35:01 UTC [25230-2] HINT:  Is another postmaster (PID 16238) running in data directory "/var/lib/postgresql/9.5/main"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.


2017-11-14 15:40:39 UTC [16238-2] LOG:  received fast shutdown request
2017-11-14 15:40:39 UTC [16238-3] LOG:  aborting any active transactions
2017-11-14 15:40:39 UTC [16240-1] LOG:  shutting down
2017-11-14 15:40:39 UTC [16240-2] LOG:  database system is shut down

-----------------------------------------------------------------
  pg_upgrade run on Tue Nov 14 15:41:07 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -b  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/tmp'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2017-11-14 15:41:07 UTC [496-1] LOG:  database system was shut down at 2017-11-14 15:40:39 UTC
2017-11-14 15:41:07 UTC [496-2] LOG:  MultiXact member wraparound protections are now enabled
2017-11-14 15:41:07 UTC [495-1] LOG:  database system is ready to accept connections
........................................................... stopped waiting
pg_ctl: could not start server
Examine the log output.


2017-11-14 18:26:00 UTC [495-2] LOG:  received fast shutdown request
2017-11-14 18:26:00 UTC [495-3] LOG:  aborting any active transactions
2017-11-14 18:26:00 UTC [497-1] LOG:  shutting down
2017-11-14 18:26:00 UTC [497-2] LOG:  database system is shut down

-----------------------------------------------------------------
  pg_upgrade run on Wed Nov 15 03:22:53 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2017-11-15 03:22:53 UTC [12182-1] LOG:  database system was interrupted; last known up at 2017-11-14 18:36:43 UTC
2017-11-15 03:22:53 UTC [12182-2] LOG:  database system was not properly shut down; automatic recovery in progress
2017-11-15 03:22:53 UTC [12182-3] LOG:  invalid record length at 0/170EAC8
2017-11-15 03:22:53 UTC [12182-4] LOG:  redo is not required
2017-11-15 03:22:53 UTC [12182-5] LOG:  MultiXact member wraparound protections are now enabled
2017-11-15 03:22:53 UTC [12178-1] LOG:  database system is ready to accept connections
........................................................... stopped waiting
pg_ctl: could not start server
Examine the log output.



-----------------------------------------------------------------
  pg_upgrade run on Wed Nov 15 03:26:53 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2017-11-15 03:26:53 UTC [21689-1] FATAL:  lock file "postmaster.pid" already exists
2017-11-15 03:26:53 UTC [21689-2] HINT:  Is another postmaster (PID 12178) running in data directory "/var/lib/postgresql/9.5/main"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.



-----------------------------------------------------------------
  pg_upgrade run on Wed Nov 15 03:29:13 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2017-11-15 03:29:13 UTC [27855-1] FATAL:  lock file "postmaster.pid" already exists
2017-11-15 03:29:13 UTC [27855-2] HINT:  Is another postmaster (PID 12178) running in data directory "/var/lib/postgresql/9.5/main"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.



-----------------------------------------------------------------
  pg_upgrade run on Wed Nov 15 05:55:38 2017
-----------------------------------------------------------------

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2017-11-15 05:55:38 UTC [20731-1] FATAL:  lock file "postmaster.pid" already exists
2017-11-15 05:55:38 UTC [20731-2] HINT:  Is another postmaster (PID 19904) running in data directory "/var/lib/postgresql/data/postgresql0"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.

Below is the output of grep data_directory

support@vrni-platform:~$ sudo grep data_directory /etc/postgresql/*/main/postgresql.conf
/etc/postgresql/10/main/postgresql.conf:data_directory = '/var/lib/postgresql/10/data/postgresql0'      # use data in another directory
/etc/postgresql/9.5/main/postgresql.conf:data_directory = '/var/lib/postgresql/data/postgresql0'        # use data in another directory

Below is the content of File Locations, Connection & Authentication settings from 10 postgresql.conf. Only the uncommented values are shown

data_directory = '/var/lib/postgresql/10/data/postgresql0'              
hba_file = '/etc/postgresql/10/main/pg_hba.conf'        
ident_file = '/etc/postgresql/10/main/pg_ident.conf'    
external_pid_file = '/var/run/postgresql/10-main.pid'   
listen_addresses = '*'         
port = 5433                    
max_connections = 100          
unix_socket_directories = '/var/run/postgresql'
ssl = on
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'

Below is the content of File Locations, Connection & Authentication settings from 9.5 postgresql.conf. Only the uncommented values are shown

data_directory = '/var/lib/postgresql/data/postgresql0'         
hba_file = '/etc/postgresql/9.5/main/pg_hba.conf'       
ident_file = '/etc/postgresql/9.5/main/pg_ident.conf'   
external_pid_file = '/var/run/postgresql/9.5-main.pid'    
listen_addresses = '*'          
port = 5432                     
max_connections = 100           
unix_socket_directories = '/var/run/postgresql' 
ssl = true                              
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'          
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'

EDIT 2

On stopping the cluster and running pg_upgrade it is failing with the same error. Looks like pg_upgrade is starting 9.5. Below is the output

ubuntu@vrni-platform:~$ sudo service postgresql status
9.5/main (port 5432): down
10/main (port 5433): down
ubuntu@vrni-platform:~$ cd /tmp/
ubuntu@vrni-platform:/tmp$ sudo service postgresql status
9.5/main (port 5432): down
10/main (port 5433): down
ubuntu@vrni-platform:/tmp$ sudo -H -u postgres /usr/lib/postgresql/10/bin/pg_upgrade \
>    -b /usr/lib/postgresql/9.5/bin \
>    -B /usr/lib/postgresql/10/bin \
>    -d /var/lib/postgresql/data/postgresql0 \
>    -D /var/lib/postgresql/10/data/postgresql0 \
>    -o ' -c config_file=/etc/postgresql/9.5/main/postgresql.conf' \
>    -O ' -c config_file=/etc/postgresql/10/main/postgresql.conf' --check

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
connection to database failed: could not translate host name "." to address: No address associated with hostname
Failure, exiting
ubuntu@vrni-platform:/tmp$ sudo service postgresql status
9.5/main (port 5432): online
10/main (port 5433): down

Below is the last few lines output of pg_upgrade_server.log

command: "/usr/lib/postgresql/9.5/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/data/postgresql0" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c config_file=/etc/postgresql/9.5/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2017-11-15 06:41:37 UTC [4556-1] LOG:  database system was interrupted; last known up at 2017-11-15 03:22:53 UTC
2017-11-15 06:41:37 UTC [4556-2] LOG:  database system was not properly shut down; automatic recovery in progress
2017-11-15 06:41:37 UTC [4556-3] LOG:  invalid record length at 0/170EB38
2017-11-15 06:41:37 UTC [4556-4] LOG:  redo is not required
2017-11-15 06:41:37 UTC [4556-5] LOG:  MultiXact member wraparound protections are now enabled
2017-11-15 06:41:37 UTC [4547-1] LOG:  database system is ready to accept connections
........................................................... stopped waiting
pg_ctl: could not start server
Examine the log output.

Solution

Firstly Thank you for listing all the steps in your process.

Can you confirm the data_directories in the postgresql.conf files?

grep data_directory /etc/postgresql/*/main/postgresql.conf

If I followed your steps correctly I had to change it thusly:

postgres@me:~$ grep data_dir /etc/postgresql/*/main/postgresql.conf
/etc/postgresql/10/main/postgresql.conf:data_directory = '/var/lib/postgresql/10/data/postgresql0'      # use data in another directory
/etc/postgresql/9.5/main/postgresql.conf:data_directory = '/var/lib/postgresql/data/postgresql0'        # use data in another directory

to get to the final output:

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Licensed under: CC-BY-SA with attribution

Not affiliated with dba.stackexchange