Here is your problem. You should be multiplying by rows
and not cols
here:
PIXEL* n = (*new)+((cols-col-1)*rows) + row;
You want to be multiplying by the width of a row in your new image, and that is the same as the number of rows in your original image.
Also, you should be swapping rows and cols here:
*newcols = rows;
*newrows = cols;
To rotate -90:
PIXEL* n = (*new)+(col*rows) + (rows-row-1);
*newcols = rows;
*newrows = cols;
To rotate 180:
PIXEL* n = (*new)+((rows-row-1)*cols) + (cols-col-1);
*newcols = cols;
*newrows = rows;
In general, the formula is:
PIXEL* n = (*new)+(newrow*newcols) + newcol;
You just need to figure out what how to determine newrow, newcols, and newcol from the previous unrotated BMP. Drawing pictures helps.