Computer Vision algorithm to tell if camera is moving?

I'm looking for a computer vision algorithm or method that can tell if the camera is moving in a video. Or maybe an alternate way of telling if the background is moving. I have a lot of videos and only want to keep the ones where the camera is stationary.

All videos feature a person talking directly to the camera (the person is mostly stationary throughout the video). I want to keep videos where the camera is stationary, but there are a lot of videos where the person is holding the camera and walking or the camera is moving around too much

I plan to run some algorithm one by one on a collection of videos to determine which video IDs have a moving camera in them.

An approach that is fast and seems to work well for me, and does not involve any fancy stuff, is to use the correlation of pixels over time as an indicator of motion:

• Make your video greyscale (lets call it video $\in \mathbb R^{T\times H\times W}$)

• Spatially Highpass filter video to make video_shp

• Temporally Lowpass the result to make video_shp_tlp

• Sample some pixels in the video (e.g. 1000). Lets call these indices $\in \mathbb N^{2 \times N}$. Only do this on the first frame and reuse the sample-indices in subsequent frames.

• Compute the correlation coefficient between the sampled pixels of the spatially-highpassed video_shp and the temporally-lowpassed video_shp_tlp: $\rho \in [0, 1] = corr($video_shp[indices], video_shp_tlp[indices])

• Take your motion_score $\in [0, 1]$ to be $1-\rho$.

Results look like this:

Python Code:

@dataclass
class MovementDetector:
    seed: int = 1234
    n_samples: int = 1000
    decay_rate: float = 0.2
    spatial_highpass_filter_width: int = 10

    _ixs: Optional[NDArray] = None
    _lowpass_img: Optional[BGRImageArray] = None

    def get_moving_score(self, image: BGRImageArray) -> float:
        """ Takes the latest image, and outputs a score indicating the 
        amount of recent motion.  Score in in [0, 1] interval, 
        with 1 being lots of motion and 0 being None.
        """
        im_grey = image.mean(axis=2)
        im_grey_highpass = im_grey - cv2.boxFilter(im_grey, ddepth=None, ksize=(self.spatial_highpass_filter_width, self.spatial_highpass_filter_width))
        im_flat = im_grey_highpass.reshape(-1)
        if self._ixs is None:
            self._ixs = np.random.RandomState(self.seed).choice(len(im_flat), size=min(len(im_flat), self.n_samples), replace=False)
            if self._lowpass_img is None:
                self._lowpass_img = np.zeros_like(im_flat)
        historical_values = self._lowpass_img[self._ixs]
        current_values = im_flat[self._ixs]
        pixel_correlation = np.corrcoef(historical_values, current_values)
        self._lowpass_img = self.decay_rate * im_flat + (1 - self.decay_rate) * self._lowpass_img
        return 1 - pixel_correlation[0, 1]

StackExchange Computer Science Q#106230, answer score: 4