Ok, I think I understand what is happening.
In order to understand the behavior, I wrote to a file the time and frame rate everytime I drew a frame. I also used a "GPU task manager" to get information on the GPU load at a specific time (GPU-Z).
When I have a lot of applications running in the back ground, my application runs fine and the GPU load is around 25%.
When I do not have a lot of applications running in the back ground, my application is constantly out of sync, the CPU load is higher and the GPU load is around not steady.
I made the observations on 2 different implementations of my code, and monitored GPU + CPU activity for 15 minutes each time.
At first, I assumed that the CPU was taking all of the load... but it seems I was wrong.