After many searches and trying to figure out ways of doing this, obviously, as soon as I post a question on the stack I figure out how to do it.
It seems that fabric uses Pointer Lock API. This means that you can do something like
var movX = e.movementX || e.mozMovementX || e.webkitMovementX || 0;
var movY = e.movementY || e.mozMovementY || e.webkitMovementY || 0;
positive numbers are right and down (respectively) negative numbers are left and down (respectively)
Hope this helps somebody in the future