So i've been using trackball controls, and as of lately i've noticed a weird bug dare i say.
If you pan and then zoom out of your new position the controls start acting very oddly, pulling the the direction of the origin (the greater the pan the more significant the issue).
I put together a fiddle which demonstrates the issue.
Simply pan away from the ball (right click + mouse move). Then zoom out.
I'm not 100% sure its not my fault, but i feel as if this example is quite simple and foolproof.
Here are some settings i have enabled:
var controls = new THREE.TrackballControls(this.camera, this.renderer.domElement);
var radius = 14;// scalar value used to determine relative zoom distances
controls.rotateSpeed = 1;
controls.zoomSpeed = 1.2;
controls.panSpeed = 1;
controls.noZoom = false;
controls.noPan = false;
controls.staticMoving = false;
controls.dynamicDampingFactor = 0.3;
controls.minDistance = radius * 1.1;
controls.maxDistance = radius * 25;
controls.keys = [ 65, 83, 68 ]; // [ rotateKey, zoomKey, panKey ]