PostgreSQL doesn't support nested sets (and keyword SETOF in this context) yet. You can use a arrays instead:
postgres=# CREATE TYPE xx AS (a int, b int);
CREATE TYPE
postgres=# CREATE TYPE yy AS (a int, b xx[]);
CREATE TYPE
postgres=# SELECT '(10,20)'::xx;
xx
---------
(10,20)
(1 row)
postgres=# select (10, ARRAY[(10,20),(30,40)]::xx[])::yy;
row
----------------------------------
(10,"{""(10,20)"",""(30,40)""}")
(1 row)
Returning some nested structures is a usually complication - almost all drivers doesn't map this structure to adequate target structure and you should to manually parser returned string. Usually the best format for returned data is scalar, vector or table.