PostgreSQL bulk import from CSV failing. ERROR: invalid input syntax for integer: “1990-01-02”

Question

I am importing data from a CSV file into a postgreSQL table.

The table looks like this:

CREATE TABLE foo_stats (
                        td                      DATE NOT NULL,
                        ins_id                  INTEGER CHECK (ins_id > 0) NOT NULL,
                        df_id                   INTEGER CHECK (df_id > 0) NOT NULL,
                        pc                      REAL NOT NULL DEFAULT 0.0,
                        ph                      REAL DEFAULT 0.0 NOT NULL,
                        pl                      REAL DEFAULT 0.0 NOT NULL,
                        av                      REAL DEFAULT 0.0 NOT NULL,
                        cv                      BIGINT DEFAULT 0 NOT NULL,
                        avi                     REAL DEFAULT 0.0 NOT NULL,
                        cmi                     BIGINT DEFAULT 0 NOT NULL,
                        vwp                     REAL CHECK (vwp >= 0) NOT NULL,
                        atr                     REAL NOT NULL DEFAULT -99999,
                        pv                      REAL NOT NULL DEFAULT -99999,
                        pr                      REAL NOT NULL DEFAULT -99999,

                        PRIMARY KEY                     (ins_id, df_id, td)
                        );

BEFORE the bulk copy, I drop the primary key using ALTER TABLE DROP CONSTRAINT PRIMARY KEY

Here is a trace of the CL when I invoke the COPY command at the psql CLI:

mydb=# COPY foo_stats FROM '/path/to/data/foo_stats.csv' WITH CSV;
ERROR:  invalid input syntax for integer: "1990-01-02"
CONTEXT:  COPY foo_stats, line 1, column id: "1990-01-02"
mydb=# 

Here is what the first few lines of the input CSV file looks like:

"1990-01-02",388,3,-99999,0.913,0.91,0.0,0,0.0,0,0,-99999,-99999,-99999
"1990-01-02",388,4,-99999,0.913,0.91,0.0,0,0.0,0,0,-99999,-99999,-99999
"1990-01-02",388,1,-99999,0.913,0.91,0.0,0,0.0,0,0,-99999,-99999,-99999
"1990-01-02",388,7,-99999,0.913,0.91,0.0,0,0.0,0,0,-99999,-99999,-99999
"1990-01-02",388,6,-99999,0.913,0.91,0.0,0,0.0,0,0,-99999,-99999,-99999

The data columns in the input file are in the same order as in the table schema. Can anyone explain why I am getting the error message?

PS: I am using PostgreSQL 8.4 on Ubuntu 10.0.4 LTS

Answer 1

Cutting and pasting your table definition and sample data here shows everything working.

So - either your table definition or your file are not as you think they are. If your table columns are in a different order, you can specify columns in the COPY command.

If the file is different, it's presumably something not visible to the naked eye, or you would have spotted it before asking. I don't suppose this file has come from a Windows machine by any chance?

Three things to check for:

1. Unnecessary byte-order-mark (BOM) at the start of the file
2. Line-endings of \r\n rather than \n
3. Any other control-codes

Try cutting + pasting your sample data and see if that imports. If so, try opening your real file in a text editor and re-saving it. If that fixes things, get a hex editor and take a look at the actual bytes in your sample file.