One possible approach:
mat = [1 2 3; 4 5 6; 7 8 9];
M = length(mat); % Assuming mat is always MxM
r = 3;
c = 1;
temp = spiral(2 * M - 1);
temp = temp(M - r + 1:end - r + 1, M - c + 1:end - c + 1);
[~, idx] = sort(temp(:));
vec = mat(idx).'
Result running with r = 3
and c = 1
vec =
7 8 4 5 6 9 1 2 3
Result running with r = 2
and c = 2
vec =
5 6 9 8 7 4 1 2 3