The problem comes from stack corruption or most probably from index out of bounds access. The fact that there are cases when your code crashes indicates there is something wrong. If your code is multi-threaded the problem may be because if actions are executed in a given order your code will try to access something out of bounds for a vector.
My advice is to run your code using valgrind and see what it will say. Usually it helps in resolving similar issues.
Also note the fact that there are cases when your code does not crash this does not mean that it works perfectly. You may still have stack corruption or similar.