How to use mutually exclusive flags in your shell and add an optional argument flag ( stuck with getopts)

Question 1

Here's how you can do it without making SID an option argument, which makes more sense to me:

die () {
    echo "ERROR: $*. Aborting." >&2
    exit 1
}

var=
opta=false
optb=false
while getopts ":ab" opt; do
  case $opt in
      a ) $optb && die "Cannot specify option a after specifying option b"
          opta=true
          ;;
      b ) $opta && die "Cannot specify option b after specifying option a"
          optb=true
          ;;
      \?) die "Invalid option: -$OPTARG. Abort"
          ;;
  esac
done
shift $(($OPTIND - 1))
test $# -eq 0 && die "You must supply SID"
test $# -eq 1 || die "Too many command-line arguments"
SID=$1

The changes I've made are:

return 1 becomes exit 1 (unrelated to your question, but die sounds like it shouldn't continue). return 1 would merely make the die function return non-successfully, but the calls to die don't check its result, it would carry on regardless.
The getopts string is :ab instead of :a:b:. No matter how you call your script, you always need to pass exactly one SID, regardless of options, so it doesn't make sense to me to include it as part of the options.
The options are stored in boolean opta and optb variables for easier checking
After all options are passed, the remaining command-line arguments are counted. Unless it's exactly one, the call to the script is invalid.

All your valid calls in the question are accepted, all the invalid ones are rejected. One note of interest about this, though: your test case c7 (-a 123 -b 123) is expected to fail, and does fail, but not because -a and -b are combined. Instead, in my approach, since -b appears after a non-option argument, it itself is a non-option argument, and the reason for rejecting it becomes "Too many command-line arguments".

Question 2

For a case where you must specify exactly one of options -s, -i or -h, and where you have two more options -v and -n which are optional, you could do something like the following:

modes=0
while getopts :vs:i:h:n opt; do
  case "$opt" in
    v)
      verbose='non-empty-string-true-flag'
      ;;
    s)
      ((modes++))
      : do something with OPTARG, the s form
      ;;
    i)
      ((modes++))
      : do something with OPTARG, the i form
      ;;
    h)
      ((modes++))
      : do something with OPTARG, the h form
      ;;
    n)
      dryrun='non-empty-string-true-flag'
      ;;
    :)
      printf 'Option -%s requires an argument\n' "$OPTARG" >&2
      usage
      exit 1
      ;;
    *)
      printf 'Invalid option: -%s\n' "$OPTARG" >&2
      usage
      exit 1
      ;;
  esac
done
shift "$((OPTIND-1))"
if [ "$modes" -eq 0 ]; then
  printf "Missing required option\n" >&2
  usage
  exit 1
elif [ "$modes" -gt 1 ]; then
  printf "Error: -h, -i and -s are mutually exclusive and may only be used once\n" >&2
  usage
  exit 1
fi

This is excerpted from a script of mine, with the processing to be done on the s, i or h arguments omitted for simplicity of illustration.

The synopsis for the script looks like:

$0 [ -v ] [ -n ] { -i IPADDRESS | -h HOSTNAME | -s HOSTSHA }

Question 3

This makes options a and b mutually exclusive:

die () {
    echo "ERROR: $*. Aborting." >&2
    exit 1
}

var=
while getopts ":a:b:" opt; do
  case $opt in
      a ) SID="$OPTARG"
          [ "$var" = 2 ] && die "Cannot specify option a after specifying option b"
          [ "$OPTARG" = b ] && die "Do not specify b as a value for option a"
          var=1
          ;;
      b ) SID="$OPTARG"
          [ "$var" = 1 ] && die "Cannot specify option b after specifying option a"
          [ "$OPTARG" = a ] && die "Do not specify a as a value for option b"
          var=2
          ;;
      :)  die "Must supply an argument to $OPTARG"
          ;;
      \?) die "Invalid option: -$OPTARG. Abort"
          ;;
  esac
done
shift $(($OPTIND - 1))
[ "$SID" ] || SID=$1
[ "$SID" ] || die "You must, at the minimum, supply SID"

Note that optargs interprets shell.sh -ab SID as option a with argument b followed by SID as an argument to the script, not part of an option. To detect that problem, the lines above involving [ "$OPTARG" = b ] && die... and [ "$OPTARG" = a ] && die... were added.

You wanted to accept shell.sh SID with no option specified. This is handled by the first two lines after the case statement.