How do I return a binary string (bytea) in a PL/Python PostgreSQL routine?

Question

I'm currently trying to write a procedure in PL/Python to perform a conversion of some data, then return the result as a bytea. (it's quite ugly, actually: marshalling the data in OCaml! Ugly in Python and OCaml at once; should I get a medal?)

Here's what it looks like:

CREATE OR REPLACE FUNCTION ml_marshal(data varchar) RETURNS bytea as $$
    import tempfile, os

    fn = tempfile.mktemp()
    f = open(fn, 'w')

    dest_fn = tempfile.mktemp()

    f.write("let outch = open_out_bin \"" + dest_fn + "\" in " +
            "Marshal.to_channel outch (" + data + ") [Marshal.No_sharing]; " +
            "close_out outch")
    f.close()

    os.system("ocaml " + fn)
    os.unlink(fn)

    f = open(dest_fn, 'r')
    res = f.read()
    f.close()

    os.unlink(dest_fn)

    return res
$$ LANGUAGE plpythonu;

In short, it writes out a small OCaml program to a tempfile which creates another tempfile with the data we want. We then read that tempfile in, destroy them both, and return the result.

Only it doesn't quite work:

meidi=# select * from tblmodel;
 modelid |      polies      
---------+------------------
       1 | \204\225\246\276
       2 | \204\225\246\276

There are four bytes in each (there should be ~130). If I stop it unlinking the files, it becomes obvious why; there are four non-NUL bytes, followed by a couple of NULs, and it appears those NULs are treated as terminators at some stage by the conversion from Python to Postgres!

Does anyone know why this happens, or how to stop it? Docs are not enlightening.

Thanks!

Edit: I found someone else with the same problem, but the solution is fairly unsatisfactory.

Answer 1

This was fixed with release 9.0. I had the same problem so I upgraded. From the release notes:

Improve bytea support in PL/Python (Caleb Welton)

Bytea values passed into PL/Python are now represented as binary, rather than the PostgreSQL bytea text format. Bytea values containing null bytes are now also output properly from PL/Python. Passing of boolean, integer, and float values was also improved.

I do not think that there is a very elegant solution for this problem in previous PostgreSQL versions.

Answer 2

You could apply another bodge- encode the return value from python as base64, and use PostgreSQL's decode function to decode it, viz. decode(ml_marshal(xxx), 'base64').

Or upgrade to 9.0 as indicated by Adrian :)