libpq call gives "UTF8": 0xe6 0x62 0x40

Question

I am a former Digital Equipment Co engineer (worked with Rdb/RTR/VMS on lots of very big systems and have 10 years of distributed systems processing experience in C years ... but quite a long time ago).

I am looking for advice on what this error means in practice, as I go about diagnosing the problem. Here is the Postgres error message

ERROR: invalid byte sequence for encoding "UTF8": 0xe6 0x62 0x40

Environment MAC Maveriks XOS, GNU C, Xcode, Postgres 9.3. (server side) with libpq

I am new to Postgres but wrote the C query set to pull all of the metadata about user tables, columns, data types, lengths, ordinal positions out of the Postgres keep them in memory in a catalogue of my own and generate all simple user table queries dynamically.

This query made it past the query preparation call.

Writing defensively I checked twice for errors:

 if ((res = PQprepare(db, statement_name, insert_query, data->nParam, NULL)) == NULL)
     Dbms_Crash(db, NULL, "Dbms_insert() PQprepare returned NULL");

 if (PQresultStatus(res) != PGRES_COMMAND_OK)
     Dbms_Crash(db, res, "Dbms_insert() Failed");

 PQclear(res);

Here is my parser dump of my generated query

insert_query:
INSERT INTO image_metadata (latitude,longitude,altitude,filename,utc_datetime)
VALUES ($1::double precision,$2::double precision,$3::double precision,$4::character varying,$5::timestamp without time zone);

Here is the call that fails:

    res = PQexecPrepared(db,
                         statement_name,
                         data->nParam,
                         (const char* const *)data->ptParam[i],
                         data->pdlParam[i],
                         data->pParamfmt,
                         PGFORMAT_STRING);

ERROR: invalid byte sequence for encoding "UTF8": 0xe6 0x62 0x40

I got past a problem with the date format in utc_datetime where Postgres expects YYYY-MM-DD HH:MM:SS for OSI universal coordinated time dates and I was feeding it YYYY:MM:DD and this one error became the next. (So my array indirection, indexing etc is working as Postgres printed the offending date, of the fifth parameter)

I am writing the obvious parameter dumper to look at what I am feeding it, but I know it got the date right when the error went away when I fixed the date format.

Is the UTF8 message coming from the filename string or one of the double precision fields?
Is passing binary a bad idea and just passing strings likely to cause less problems?

My schema will eventually have say 30 tables and this type of error in memory alignment on the server side is very worrying. My aim is to write zero SQL code.

Answer 1

In your PQprepare() function the last parameter is NULL. That implies that all query parameters are in text format. If you passed some of them as binary, you sent just a garbage. My advice is to use text format for parameters in prepared queries.