You LastIndexOf parameters are messed up, as well as the Length of the selection, where you need to substract the starting point in order to get the proper length.
Try a simpler version:
int textStart = mRtbxOperations.Text.LastIndexOf(@"{",
mRtbxOperations.SelectionStart);
if (textStart > -1) {
int textEnd = mRtbxOperations.Text.IndexOf(@"}", textStart);
if (textEnd > -1) {
mRtbxOperations.Select(textStart, textEnd - textStart + 1);
mRtbxOperations.SelectionBackColor = Color.LightBlue;
}
}