PostgreSQL 9.6 column dropping and side-effects on SQL functions with CTEs

Question

If I had a table with 3 columns - say A, B and D - and I had to introduce a new one - say C to replace the current position of D. I would use the following method:

1. Introduce 2 new columns as C and D2.
2. Copy the contents of D to D2.
3. Delete D.
4. Rename D2 to D.

The new order would be A, B, C and D.

I thought this was a legitimate practice as (so far) it produced no issues.

However, today I came across a problem when a function carrying out a statement on the same table returned the following error:

table row type and query-specified row type do not match

And the following detail:

Query provides a value for a dropped column at ordinal position 13

I tried restarting PostgreSQL, doing a VACUUM FULL and finally deleting and re-creating the function as suggested here and here but these solutions did not work (aside from the fact that they try tackling a situation where a system table has been altered).

Having the luxury of working with a very small database I exported it, deleted it and then re-imported it and that fixed the issue with my function.

---

I was aware of the fact that one should not mess around with the natural order of columns by modifying system tables (getting hands dirty with pg_attribute, etc.) as seen here:

Is it possible to change the natural order of columns in Postgres?

Judging by the error thrown by my function I now realize that shifting the order of columns with my method is also a no-no. Can anyone shine some light as to why what I am doing is also wrong?

---

Postgres version is 9.6.0.

Here is the function:

CREATE OR REPLACE FUNCTION "public"."__post_users" ("facebookid" text, "useremail" text, "username" text) RETURNS TABLE (authentication_code text, id integer, key text, stripe_id text) AS '

-- First, select the user:
WITH select_user AS
(SELECT
users.id
FROM
users
WHERE
useremail = users.email),

-- Second, update the user (if user exists):
update_user AS
(UPDATE
users
SET
authentication_code = GEN_RANDOM_UUID(),
authentication_date = current_timestamp,
facebook_id = facebookid
WHERE EXISTS (SELECT * FROM select_user)
AND
useremail = users.email
RETURNING
users.authentication_code,
users.id,
users.key,
users.stripe_id),

-- Third, insert the user (if user does not exist):
insert_user AS
(INSERT INTO
users (authentication_code, authentication_date, email, key, name, facebook_id)
SELECT
GEN_RANDOM_UUID(),
current_timestamp,
useremail,
GEN_RANDOM_UUID(),
COALESCE(username, SUBSTRING(useremail FROM ''([^@]+)'')),
facebookid
WHERE NOT EXISTS (SELECT * FROM select_user)
RETURNING
users.authentication_code,
users.id,
users.key,
users.stripe_id)

-- Finally, select the authentication code, ID, key and Stripe ID:
SELECT
*
FROM
update_user
UNION ALL
SELECT
*
FROM
insert_user' LANGUAGE "sql" COST 100 ROWS 1
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER

I performed the renaming/reordering on both column facebook_id and stripe_id (a new column was added before these, which is the reason for the renaming, but is not touched by this query).

Having the columns in a certain order is purely out of interest for order. However, the reason for asking this question is out of concern that a simple renaming and deleting of a column may trigger real issues for somebody using functions in production mode (as happened to myself).

Answer 1

Probable bug on 9.6 and 9.6.1

This completely looks like a bug to me...

I don't know why it happens, but I can confirm that it happens. This is the simplest found setup that reproduces the problem (in version 9.6.0 and 9.6.1).

CREATE TABLE users
(
    id SERIAL PRIMARY KEY,
    email TEXT NOT NULL,
    column_that_we_will_drop TEXT
) ;

-- Function that uses the previous table, and that has a CTE
CREATE OR REPLACE FUNCTION __post_users
    (_useremail text) 
RETURNS integer AS
$$
-- Need a CTE to produce the error. A 'constant' one suffices.
WITH something_even_if_useless(a) AS
(
    VALUES (1)
)
UPDATE
    users
SET
    id = id
WHERE 
    -- The CTE needs to be referenced, if the next
    -- condition were not in place, the problem is not reproduced
    EXISTS (SELECT * FROM something_even_if_useless)
    AND email = _useremail
RETURNING
    id
$$
LANGUAGE "sql" ;

After this setup, the next statement just works

SELECT * FROM __post_users('a@b.com');

At this point, we DROP one column:

ALTER TABLE users 
    DROP COLUMN column_that_we_will_drop ;

This change makes the next statement to generate an error

SELECT * FROM __post_users('a@b.com');

which is the same as mentioned by @Andy:

ERROR: table row type and query-specified row type do not match
SQL state: 42804
Detail: Query provides a value for a dropped column at ordinal position 3.
Context: SQL function "__post_users" statement 1
    SELECT * FROM __post_users('a@b.com');

Dropping and recreating the function does NOT solve the problem.
VACUUM FULL (the table or the whole database) does not solve the problem.

---

The bug report was passed to the appropriate PostgreSQL mailing list and we had a very fast response:

I can't reproduce this in HEAD or 9.6 branch tip. I believe it was already fixed by this patch, which went in a bit after 9.6.1:

https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=f4d865f22

But thanks for the report!

regards, tom lane

---

Version 9.6.2

On 2017-03-06, I can confirm that I cannot reproduce this behaviour on version 9.6.2. That is, the bug seems to have been corrected on this release.

UPDATE

Per comment of @Jana: "I can confirm the bug is present in 9.6.1 and was fixed in 9.6.2. The fix is also listed on postgres release website: Fix spurious "query provides a value for a dropped column" errors during INSERT or UPDATE on a table with a dropped column"

---

Answer 2

I know you've probably heard this before, but this is a horrible idea.

• Logical ordering only effects things like SELECT *
• The effect of logical ordering is limited to appearance.

So if not mattering at all doesn't dissuade you and we acknowledge that we're just playing Photoshop with row structure and obsessing over display, let's explain some more things.

• Logical ordering doesn't exist in PostgreSQL
• Physical ordering has real benefits (table packing)
• PostgreSQL doesn't give you any control over Physical Ordering after CREATE TABLE either (though it would be a much higher priority)

So PostgreSQL is a bad display layer. Onto of all that, while ALTER works fine you shouldn't temp the dragon. Both ALTER TABLE, and the CAVEAT section make mention of this,

Some DDL commands, currently only TRUNCATE and the table-rewriting forms of ALTER TABLE, are not MVCC-safe. This means that after the truncation or rewrite commits, the table will appear empty to concurrent transactions, if they are using a snapshot taken before the DDL command committed. This will only be an issue for a transaction that did not access the table in question before the DDL command started — any transaction that has done so would hold at least an ACCESS SHARE table lock, which would block the DDL command until that transaction completes. So these commands will not cause any apparent inconsistency in the table contents for successive queries on the target table, but they could cause visible inconsistency between the contents of the target table and other tables in the database.

And, if all that isn't enough, and you still want to pretend that this is a good idea rather a horrible idea. Then try this,

1. Dump the table with pg_dump -t
2. Reorder the columns by hand in the dump.
3. Load the stuff into a TEMP table.
4. BEGIN a transaction
5. DROP the old table entirely,
6. RENAME the temp table to the prod table.
7. COMMIT

If all of that sounds excessive, keep in mind that updating rows in the database requires rewriting the rows (assuming they're not TOAST. You're having to parse the data and rebuilt table schema, but either way you've got to rewrite the row. If I had to do this task, that's how I would do it.

But, all this is speaking generally. No one has reproduced your results.

You've given a test-case that we can't run

ERROR:  column users.authentication_code does not exist
LINE 24: users.authentication_code,

And, you haven't told us the exact version you're on.

Answer 3

I got around this bug by backing up and restoring my database.

Steps for Heroku

• Turn on maintenance mode: heroku maintenance:on
• Back up database: heroku pg:backups:capture
• Restore database: heroku pg:backups:restore
• Restart app: heroku restart
• Turn off maintenance mode: heroku maintenance:off

Answer 4

I ran into this bug too. For those who do not want to fully backup / restore their DB. Know that simply copying the table works. There is no "magical" way to copy a table though. I did it by using :

SELECT * INTO mytable_copy FROM mytable;
ALTER TABLE mytable RENAME TO mytable_backup; -- just in case. you never know
ALTER TABLE mytable_copy RENAME TO mytable;

After that, you will still need to manually recreate your indexes, foreign keys and defaults. Recreating the table like this made the bug disappear.