Casting a ray from mouse through distortion matrix

Question

I've searched the board, as well as the oculus board, and unity board. Couldn't really find something that helped.

I'm working on a vehicle simulation. Before we started using the oculus, it was just a regular first person perspective. You used a racing wheel/pedals to drive and the mouse to control all the buttons and switches etc. We use raycasting from the mouse point on the screen into the world to interact with the various controls in the vehicle.

Now that we're using the oculus, the raycast isn't taking into account the distortion matrix used on the oculus cameras. So you're not actually casting a ray at what you're visually clicking on. Using Debug.DrawRay I found that it was slightly off. Just to be sure, I disabled the lens correction via inspector on the OVRCameraController and sure enough the raycasting was working again.

The ray itself is calculated the usual way one does when firing from the mouse point: ScreenPointToRay(Input.mousePosition);

Would anyone have any idea how I can adjust my ray so it works with lens correction on?

Cheers, Gordon

Answer 1

Alright, what I ended up doing was creating a 3D cursor, bypassing the distortion matrix entirely.

I placed a gameobject at the same place as the "head" (between left eye and right eye cameras). It has a script on it that rotates up/down/left/right based on mouse movement. I then temporarily put a spot light with a narrow cone and high intensity on it so it looked like a laser pointer. I figured if the light is hitting things, so should a raycast of the same origin. Which ended up working.

However this didn't really solve the issue of using a cursor. I tried a number of things that ultimately didn't work (didn't line up with with where the light/raycast hit).

Finally I realized I was overlooking something very simple. I lowered the near clipping plane of the cameras and placed a plane as close as I could to the camera while still being visible. I then rotated it on local y by 180 so it would be invisible to the cameras and not block ray casts.

I then added some code so that when a raycast hit something, it would fire a second raycast from the hit point back to the origin. On the way it would have to hit the plane, which was essentially at the near clipping plane. I would then move my 3D cursor to that hit point.

Now it works as intended. Where the cursor is, is where the original raycast hit. The cursor now matched the position of the laser dot. So then I removed the light component. Done.

Hope this helps someone else someday.

Answer 2

Simply multiply the Distortion-Matrix with the Ray's Vectors (Position and Normal) and you have your new Ray. I would suggest using Homogeneous Coordinates with 4x4 Matrix and Vec4's where Positions has component w = 1.0 and Normals have w = 0.0; This way you can simply multiply and you are done - depending on lookup direction you might have to use the inverse matrix :)