You can't code that logic in a constraint.
You can create a unique function-based index, however, which probably lets you accomplish the same task
CREATE UNIQUE INDEX idx_my_table
ON my_table ( CASE WHEN a <= b THEN a ELSE NULL END,
CASE WHEN a <= b THEN b ELSE NULL END,
CASE WHEN a <= b THEN c ELSE NULL END );
This takes advantage of the fact that Oracle doesn't store completely NULL
values in the index structure. This allows you to have as many rows where b < a as you'd like and none of them would be stored in the index structure.