Once you subtract the background, you should threshold the resulting image to create a binary foreground mask. Set all the differences less than a threshold to 0 (background), and set the ones greater than or equal to a threshold to 1 (foreground). You can then use morphology such as imopen
to get rid of small noisy specks in the background and imclose
to get rid of small gaps or holes in the foreground.
Once you are happy with your foreground mask, you can use it as a logical index to set the background pixels to whatever color you choose.