Okay, from what I deduced, I would do this:
\{number(?:\((\-?\d+)(?:\,(\-?\d+))?(?:\,(\-?\d+))?\))?\}
Then carry out operations on the captured groups to valid the range of the integers such as...
[Pseudo code since I don't know what language you are using]
captured integers = "capture1", "capture2", "capture3"
if{("capture1" < "capture2" && "capture1" > "capture3") ||
("capture1" > "capture2" && "capture1" < "capture3")} {
Do something
} else {
Do something else; like reject or throw error
}