Streaming H264 video from PiCamera to a JavaFX ImageView

I'm currently working on a robotics application where a video feed is being displayed from a Raspberry Pi 3.

I've been working on a way to stream the video directly into JavaFX (the rest of the UI is created in this), however, my knowledge of video streaming is very limited. The goal for the video system is to maintain decent video quality and FPS while reducing latency as much as possible (looking for sub 100 ms). H264 video was chosen as the format for it's speed, but I hear that sending raw video could be faster as there is no compression (could not get raw video to work well at all).

Running my code I am capable of streaming a Pi camera at about 120-130ms of latency and ~48 frames per second. I would like to continue to reduce the latency of this application, and would like to make sure that I'm making decisions for the correct reasons.

The largest issue I have so far is start-up time; it takes about 15-20 seconds for the video to initially launch and catch up to the latest frame.

The following code is an MCVE of the video system. If anyone is interested in reproducing this, you can get it running on a Raspberry Pi (mine is a Raspberry Pi 3) with python-picamera installed. You'll also need a Java Client with JavaCV installed. My version info is org.bytedeco:javacv-platform:1.3.2.

Python side:

We decided to use a Python library to control the video stream because it provides a nice wrapper around the picamera command-line tool. The output from the video is being sent over a TCP connection and will be received by a Java client. (The way we remotely launch this application has been left out of the review because I just wanted this post to focus on the video aspects)

`import picamera
import socket
import signal
import sys

with picamera.PiCamera() as camera:
camera.resolution = (1296, 720)
camera.framerate = 48

soc = socket.socket()
soc.connect((sys.argv[1], int(sys.argv[2])))
file = soc.makefile('wb')

try:
camera.start

Architectural Ideas

Let's start at the Architecture of your Application and the data transfer.
There's basically two places where we can optimize the performance of your application. I'm ignoring latency for now, since that is mostly determined by the network and the performance of the image processing.

This means if we can improve the speed of image processing, latency will also go down. That's the first component.

The second component is to make the network transfer less vulnerable to overhead and stalling. A great recap of TCP versus UDP is this joelonsoftware article.

Taking the information there into account, I'd think you're better off sending your video over UDP.

Performance review

It's problematic to require a full rendering of an image to update the image-view. Most realtime rendering applications use the idea of a frame-/backbuffer. What happens there is the following:

• An image is rendered to the backbuffer.

• When it's finished, the framebuffer and the backbuffer are swapped

• The next image is rendered to the backbuffer while the framebuffer is displayed.

As far as I can tell, you're missing out on a lot of performance by ineffectively handling how data is passed between network and image-view. It'd help to see what Java2DFrameConverter does exactly.

Code style review

There's a few things that struck me as odd in your code. The following is a review without taking the performance into account directly:

streamToImageView takes a lot of arguments. You can drastically reduce their number by partially applying them outside of the method. Additionally the converter can be a static field, though I can understand if you want to have an instance for every invocation of the method.

This might also be the place for the backbuffer idea, since you can reuse Image instances when rendering. I'm not sure, but you might be able to just set the Image to the imageView once and then reuse the already set instance.

The method then looks like this:

public static void streamImageToView(ImageView view, int port, int socketBacklog, Consumer grabberSettings) {
    try (final ServerSocket server = new ServerSocket(port, socketBacklog);
         final Socket clientSocket = server.accept();
         final FrameGrabber grabber = new FFmpegFrameGrabber(clientSocket.getInputStream());
    ) {
        grabberSettings.accept(grabber);
        grabber.start();
        while (!Thread.interrupted()) {
            final Frame frame = grabber.grab();
            if (frame == null) {
                continue;
            }
            final BufferedImage bufferedImage = converter.convert(frame);
            if (bufferedImage != null) {
                Platform.runLater(() -> {
                    SwingFXUtils.toFXImage(bufferedImage, view.getImage());
                    view.setImage(view.getImage()); 
                    // might not be required. Forces repaint
                });
            }
        } catch (IOException ex) {
            // same as before
        }
    } 
}

I like very much that you're making all the variables final wherever possible.
FWIW I leave the partial application of the streamToImageView arguments to you as an exercise ;)

StackExchange Code Review Q#163042, answer score: 5