Upsert in Amazon RedShift without Function or Stored Procedures

Question

As there is no support for user defined functions or stored procedures in RedShift, how can i achieve UPSERT mechanism in RedShift which is using ParAccel, a PostgreSQL 8.0.2 fork.

Currently, i'm trying to achieve UPSERT mechanism using IF...THEN...ELSE... statement e.g:-

IF NOT EXISTS(SELECT...WHERE(SELECT..))
              THEN INSERT INTO tblABC() SELECT... FROM tblXYZ
              ELSE UPDATE tblABC SET.,.,.,. FROM tblXYZ WHERE... 

which is giving me error. As i'm writing this code independently without including it in function or SP's. So, is there any solution to achieve UPSERT.

Thanks

Answer 1

You should probably read this article on upsert by depesz. You can't rely on SERIALIABLE for this since, AFAIK, ParAccel doesn't support full serializability support like in Pg 9.1+. As outlined in that post, you can't really do what you want purely in the DB anyway.

The short version is that even on current PostgreSQL versions that support writable CTEs it's still hard. On an 8.0 based ParAccel, you're pretty much out of luck.

I'd do a staged merge. COPY the new data to a temporary table on the server, LOCK the destination table, then do an UPDATE ... FROM followed by an INSERT INTO ... SELECT. Doing the data uploads in big chunks and locking the table for the upserts is reasonably in keeping with how Redshift is used anyway.

Another approach is to externally co-ordinate the upserts via something local to your application cluster. Have all your tools communicate via an external tool where they take an "insert-intent lock" before doing an insert. You want a distributed locking tool appropriate to your system. If everything's running inside one application server, it might be as simple as a synchronized singleton object.