please don't store pointers to Mat in your vector(or anywhere else !).
those things are refcounted internally, like smartpointers, and you're wrecking that by storing/copying pointers (a vector of pointers to smartpointers would sound silly anyway, no?).
use a plain vector<Mat>
, trade some additional ~56 bytes per item against sound sleep tonight ;)