postgresql - Schedule Insert data from dblink to a local table In a

Question

I am trying to link a remote database to a local database using dblink. What want to achieve here is:

1. I want to be fetching only data from the latest row(s) every 10 secs from a table in remote database.

2. I wish to insert the data into a local database into a pre-existing table. In this case, I wish to insert the data I collected from the remote database, plus some others like primary key, and some sequenses which I dont get from the remote database into the table.

Any suggestions would be appreciated.

Answer 1

• You'd need an indexed column in remote table that will increase every time a row will be inserted or updated there (not a timestamp, as many rows can have the same timestamp, a computer clock can sometimes go backwards etc.). If you don't ever update then a serial primary key would suffice (enforce that updates are not allowed with a trigger if you decide to rely on this). Deletes would also not be synchronized, so I'd recommend to enforce that they're not allowed using trigger too.

• You'd need a cron (or scheduler on Windows) job that will connect to your database and perform synchronization, as PostgreSQL has no mechanism for periodic tasks.

• This job would just do:

  start transaction;
  lock table local_tablename in exclusive mode;
  dblink_connect(…);
  insert into local_tablename (id, data, row_counter)
    select * from dblink(
      'select id, data, row_counter from remote_tablename
         where row_counter>'||(select coalesce(max(row_counter),-1) from local_tablename)
    ) as t(id int, data text, row_counter int);
  commit;

  It needs to be in transaction and protected with a lock, because it can break if it would run concurrently with another synchronization job (for example if previous job took more than 10 seconds).

  `coalesce` is needed in case `local_tablename` has no rows yet - it would then not insert anything without it. It assumes that always `row_counter>=0`.