Did you try to remove the space lines from the properties definition?
WARNING - Property "JdbcUsername " not supported by "customDS"
WARNING - Property "JdbcPassword " not supported by "customDS"
WARNING - Property "JdbcUrl " not supported by "customDS"
WARNING - Property "JdbcDriver " not supported by "customDS"
p.put("customDS.JdbcDriver ", "org.hsqldb.jdbc.JDBCDriver");
p.put("customDS.JdbcUrl ", "jdbc:hsqldb:file:target/custom");
p.put("customDS.JdbcUsername ", "sa");
p.put("customDS.JdbcPassword ", "");
I think this should work (without the spaces).
Hope this helps.
EDIT
Maybe its your property defining the datasource:
p.put("customDS", "new://Resource?type=javax.sql.DataSource");
The documentation specifies to look something like this:
p.put("customDS = new://Resource?type", "DataSource");
EDIT
Right properties to use are listed here : http://tomee.apache.org/containers-and-resources.html
p.put("customDS.JdbcDriver", "org.hsqldb.jdbc.JDBCDriver");
p.put("customDS.JdbcUrl", "jdbc:hsqldb:file:target/custom");
p.put("customDS.UserName", "sa");
p.put("customDS.Password", "");