using orientation sensor data to predict image points

I want to use Android's gyro/accelerometer/magnetometer to predict (over a very short time interval) how image points corresponding to a fixed object will change (without trying to scan for them in the image). In particular, suppose we have an object of known size with model points $v_1, \dots, v_N$. In this question, all vectors are assumed to be in homogenous form.

Here is what I am trying:

-
Detect the object. Let the pixel coordinates of the model points be denoted by $u_1, \dots, u_N$. Given the calibration matrix $K$ and negligible lens distortion, solve the PnP problem to get rotation $R_0$ and translation $t_0$ from the coordinates in which the vectors $v_1, \dots, v_N$ are defined into camera-centric coordinates, shown in black in the image above.
In particular, the solution to the PnP problem solves the following equation.
$$
u_i = K \begin{bmatrix} R_0 & t_0 \\ 0^\top & 1\end{bmatrix} v_i.
$$

-
Simultaneously query the Android rotation sensor for a rotation matrix.

This is the relative orientation from Earth coordinates to the device coordinates (colored frame in the picture above).
The direction of the this transformation is discussed informally in the Android developer site:


The rotation vector represents the orientation of the device as a combination of an angle and an axis, in which the device has rotated through an angle $\theta$ around an axis ($x$,$ y$, or $z)$.

Let $R_1$ denote this rotation matrix.

-
Rotate the phone a small amount and query the rotation sensor again for $R_2$.

-
Render the predicted location to the pixel coordinates. For this, lets assume the sensors and the camera are collocated, so that the transformation from camera coordinates to device coordinates is a pure rotation:
$$
D=
\begin{bmatrix}
0 & -1 & 0 & 0\\
-1 & 0 & 0 & 0 \\
0 & 0 & -1 & 0 \\
0 & 0 & 0 & 1
\end{bmatrix}
$$
This matrix maps the black frame to the colorful frame in the picture.
Note that the $+z$ axis of the camera frame (black in the pi

It turns out that I had two bugs that broke my experimental results:

• Even though the rotation matrix reports the rotation from Earth to Device, I should have been applying the opposite rotation to the point. It took me a day to convince myself of this. Basically, if you rotate the whole frame about, e.g., $+x$, then coordinates that remained fixed w.r.t. the device will rotate w.r.t. the Device frame about $-x$.

• I had a wrong minus sign in the rotation about the y-axis in the experimental results that I reported at the end. Note that y-axis rotations have a different +/- pattern than x- and z- axis rotations. See wikipedia for basic rotations.

The method described herein works fine (at least in theory). I'll post an update after I port my python dummy code into Android and see how it works.

StackExchange Computer Science Q#76246, answer score: 3