In your description, you are planning to use the machine learning method for traffic sign recognition. However, there are more things should be known before that.
Object recognition(Ok, here the object is Sign especially), can be divided into two main parts: Detection and Recognition. For the detection part you may need the color threshold and polygon detection to separate the sign from the complex environment.(Which has so less relationship with the positive or negative training data.)
And for the recognition part, Machine learning is also just one option. As far as I know, you can use sift or SURF for this part. And there are some papers about that. If you decided to use the training data for recognition, please post a question then.
All in all, please read some related papers from google.
Good luck!