Finally found the solution from a mixture of matrices and quaternions:
//Setup
var ux = new THREE.Vector3(1,0,0);
var uy = new THREE.Vector3(0,1,0);
var uz = new THREE.Vector3(0,0,1);
var direction = ux.clone();
var m4 = new THREE.Matrix4();
var dq = new THREE.Quaternion(); //direction quad base
var dqq; //final direction quad
var dq2 = new THREE.Quaternion();
dq2.setFromAxisAngle(uz,Math.PI/2); //direction perpendicular rot
//Update
if (velocity.length() < 0.1) return;
if (velocity.x) { focalPoint.translateY( velocity.x ); }
if (velocity.y) { focalPoint.translateX( velocity.y ); }
//create new direction from focalPoint quat, but perpendicular
dqq = dq.clone().multiply(focalPoint.quaternion).multiply(dq2);
velocity.multiplyScalar(dropOff);
//forward direction vector
direction = ux.clone().applyQuaternion(dqq).normalize();
//use Matrix4.lookAt to align focalPoint with the direction
m4.lookAt(focalPoint.position, planet.mesh.position, direction);
focalPoint.quaternion.setFromRotationMatrix(m4);
var cameraOffset = relativeCameraOffset.clone();
cameraOffset.z = cameraDistance;
cameraOffset.applyQuaternion(focalPoint.quaternion);
camera.position = focalPoint.position.clone().add(cameraOffset) ;
//use direction for camera rotation as well
camera.up = direction;
camera.lookAt( focalPoint.position );
This is the hard core of it. It pans (and with some extension rotates) around the planet without the poles being an issue.