you can use radio button group and add one more option ("no selection") or you can use radio button group with 3 options and add button to set all option to off
if you want to use checkbox it works when you write callbacks for each checkbox instead of uipanel (I'm affraid callback for uipanel is not evaluated at all) (here I assume that in MyUipanel there are only checkboxes):
function MyCheckboxCallback(hObject, eventdata, handles)
other = setdiff(get(handles.MyUipanel,'Children'),hObject);
for ii = 1:length(other)
set(other(ii),'Value',get(other(ii),'Min'));
end
function MyCheckbox1_Callback(hObject, eventdata, handles)
MyCheckboxCallback(hObject, eventdata, handles)
function MyCheckbox2_Callback(hObject, eventdata, handles)
MyCheckboxCallback(hObject, eventdata, handles)
function MyCheckbox3_Callback(hObject, eventdata, handles)
MyCheckboxCallback(hObject, eventdata, handles)