How to insert (file) data into a PostgreSQL bytea column?
-
16-10-2019 - |
Question
This question is not about bytea v. oid v. blobs v. large objects, etc.
I have a table containing a primary key integer
field and a bytea
field. I'd like to enter data into the bytea
field. This can, presumably, be done by one of the PL/
languages, and I may look into doing this with PL/Python
in the future.
As I am still testing and experimenting, I would simply like to insert data from a file (on the server) using "standard" SQL statements. I am aware that only administrators with write permission on the server would be able to insert data in the way I would like to. I'm not concerned about that at this stage as users would not be inserting bytea
data at present. I have searched the various StackExchange sites, the PostgreSQL Archives and the Internet generally, but have not been able to find an answer.
Edit: This discussion from 2008 implies that what I want to do is not possible. How are bytea
fields used then?
Edit: This similar question from 2005 remains unanswered.
Solved: The details provided here on the psycopg
website provided the basis for a solution I've written in Python. It may also be possible to insert binary data into a bytea
column using PL/Python
. I don't know if this is possible using "pure" SQL.
Solution
as superuser:
create or replace function bytea_import(p_path text, p_result out bytea)
language plpgsql as $$
declare
l_oid oid;
begin
select lo_import(p_path) into l_oid;
select lo_get(l_oid) INTO p_result;
perform lo_unlink(l_oid);
end;$$;
lo_get
was introduced in 9.4 so for older versions you would need:
create or replace function bytea_import(p_path text, p_result out bytea)
language plpgsql as $$
declare
l_oid oid;
r record;
begin
p_result := '';
select lo_import(p_path) into l_oid;
for r in ( select data
from pg_largeobject
where loid = l_oid
order by pageno ) loop
p_result = p_result || r.data;
end loop;
perform lo_unlink(l_oid);
end;$$;
then:
insert into my_table(bytea_data) select bytea_import('/my/file.name');
OTHER TIPS
Use pg_read_file('location_of file')::bytea
.
For example,
create table test(id int, image bytea);
insert into test values (1, pg_read_file('/home/xyz')::bytea);
This solution isn't exactly efficient in terms of runtime, but it's trivially easy compared to making your own headers for COPY BINARY
. Further, it doesn't require any libraries or scripting languages outside of bash.
First, convert the file into a hexdump, doubling the size of the file. xxd -p
gets us pretty close, but it throws in some annoying newlines that we have to take care of:
xxd -p /path/file.bin | tr -d '\n' > /path/file.hex
Next, import the data in PostgreSQL as a very large text
field. This type holds up to one GB per field value, so we should be okay for most purposes:
CREATE TABLE hexdump (hex text); COPY hexdump FROM '/path/file.hex';
Now that our data is a gratuitously large hex string, we use PostgresQL's decode
to get it into a bytea
type:
CREATE TABLE bindump AS SELECT decode(hex, 'hex') FROM hexdump;
The answer with xxd is nice and, for small files, very fast. Below is an example script that I'm using.
xxd -p /home/user/myimage.png | tr -d '\n' > /tmp/image.hex
echo "
-- CREATE TABLE hexdump (hex text);
DELETE FROM hexdump;
COPY hexdump FROM '/tmp/image.hex';
-- CREATE TABLE bindump (binarydump bytea);
DELETE FROM bindump;
INSERT INTO bindump (binarydump)
(SELECT decode(hex, 'hex') FROM hexdump limit 1);
UPDATE users
SET image=
(
SELECT decode(hex, 'hex')
FROM hexdump LIMIT 1
)
WHERE id=15489 ;
" | psql mydatabase
Use the Postgres COPY BINARY function. This is broadly equivalent to Oracle's external tables.