If you want to get the selected rows in a button-callback, there's no need for the CellSelection/CellEditCallback.
As I suggested in my first comment, simply get the data and find the selected rows:
function button_callback(hObject, evt, handles)
% get the data - identical to setting the data
data = get(handles.tableHandle, 'Data');
checkBoxColumn = 4;
% logical indices of selected rows
isRowSelected = [data{:, checkBoxColumn}];
% if you want the indices
idxSelectedRows = find(isRowSelected);
end