I think you mix the concept of moving the camera and rotating your object. First you need to separate the action of rotating your object from any interaction with the camera. You should be able to do anything with the object without changing the camera at all. To do this just use the model matrix to describe the translation/rotation/scaling of your object. Then you handle your camera separatly by using Matrix.setLookAtM with an eyeX, eyeY, eyeZ that is independent of your model position. And a point to look at that in your case is dependent of the eyeX, eyeY, eyeZ, as you want to be looking at the same direction all the time.
For example, if you want to rotate your object 90 degrees around the Y axis, use:
Matrix.rotateM(mModelMatrix, 0, 90, 0.0f, 1.0f, 0.0f);
Also remember that all matrix operations after Matrix.setIdentityM will be done in a backwards order, ie if you rotate first and then translate as you do the first thing in your code, you will first translate your object away from its origin position, then rotate around the origin, making your object do a kind of rotation around of the first origin, instead of the center of your object.
Then lets say you have translated your camera 100 units to the left and looking at -Z direction, use:
Matrix.setLookAtM(mViewMatrix, offset, -100.0f+camX, camY, camZ, -100.0f+camX, camY, -1.0f+camZ, upX, upY, upZ);