You can do this with a function-based index:
create unique index table1_col1_col2 on table1(least(col1, col2), greatest(col1, col2));
Alternatively, you could add a constraint that col1 < col2
and create a unique index on col1, col2
. You could enhance this by adding an insert/update trigger to swap the values when given in the "wrong" order.