Edited
My first answer was based on a misunderstanding of the question. I have included revised code. It now appears to be much more like your original. I get the result
1.0e+003 * [0.6475 1.6242]
for the transformed point. When plotted, this appears (as expected) almost perfectly reflected across the center.
Working code
clear all
close all
%% Points and angles (angle in radians)
angle = -3.1150
mypoint = [634 232] % Near top, middle
%% Sample data
load('mandrill', 'X', 'map');
im = uint8(X);
% Using Image Processing Toolkit to create an image 1300 wide by 1856 high
im = imresize(im, [1856 1300]);
%% Calculate 'center' from extrema
m = size(im,2)/2 % width/2 (I want to rotate around the center of the image)
n = size(im,1)/2 % height/2
%% Tx matrices
first = [...
1 0 -m;
0 1 -n;
0 0 1];
third = [...
1 0 m;
0 1 n;
0 0 1];
second = [...
cos(angle) -sin(angle) 0;
sin(angle) cos(angle) 0;
0 0 1];
%% Use homogenous coords
mp_hom = [mypoint 1]
%% Calculate (note because we premultiply,
rotated_hom = third* second* first* mp_hom';
rotated = rotated_hom'
%% Show it!
imshow(im)
hold on
plot(mypoint(:,1), mypoint(:,2), 'g+', 'MarkerSize', 12, 'LineWidth', 3)
plot (m, n, 'b*', 'MarkerSize', 12, 'LineWidth', 3)
plot(rotated(:,1), rotated(:,2), 'gx', 'MarkerSize', 12, 'LineWidth', 3)