If stamp size is too small compared to the main image, using SURF
descriptor alone may not be a feasible choice.
Since you have a finite number of stamp reference images, you can try template matching
and after detecting the stamp region and calculating the orientation, you can just calculate the correlations with references and select the highest correlation as the detected object.
Template matching operation can be applied on each RGB channel (assuming color image) and matching scores can be summed as a final decision. However, the harder part is the detection of stamps and their orientations.
You can also use SURF keypoints on 3 RGB channels separately in order to utilize color information. After extracting keypoints for each channel, you can calculate matching scores separately and sum the three scores to obtain a final decision.
As another feature, you can use calculate color histograms
of reference images and during testing, you can commpare these histograms to find a good match. THis feature is rotation independent and simple to calculate.