AndreyFilippov committed changes to the Elphel git project :
working on expansion + bug fixes

Oleg Dzhimiev committed changes to the Elphel git project :
thumbs and list in popup

Oleg Dzhimiev committed changes to the Elphel git project :
minor

Oleg Dzhimiev committed changes to the Elphel git project :
kml restored message

Oleg Dzhimiev committed changes to the Elphel git project :
kml saving results display

Oleg Dzhimiev committed changes to the Elphel git project :
kml save with error codes

Oleg Dzhimiev committed changes to the Elphel git project :
fixed kml saving

Oleg Dzhimiev committed changes to the Elphel git project :
scrolling + dragging the list

Oleg Dzhimiev committed changes to the Elphel git project :
edit mode on/off and errors displaying

Oleg Dzhimiev committed changes to the Elphel git project :
alignment code clean up, todo: error reporting and mode on/off by checkbox

Oleg Dzhimiev committed changes to the Elphel git project :
+ apply dialog, not finished

Oleg Dzhimiev committed changes to the Elphel git project :
Gauss-Newton algorithm to align heading and lat,lng

AndreyFilippov committed changes to the Elphel git project :
second expand pass, over possible background

AndreyFilippov committed changes to the Elphel git project :
working on initial z-map generation

Fig.1 Live stream latency testing

Recently we had an inquiry whether our cameras are capable of streaming low latency video. The short answer is yes, the camera’s average output latency for 1080p at 30 fps is ~16 ms. It is possible to reduce it to almost 0.5 ms with a few changes to the driver.

However the total latency of the system, from capturing to displaying, includes delays caused by network, pc, software and display.

In the results of the experiment (similar to this one) these delays contribute the most (around 40-50 ms) to the stream latency – at least, for the given equipment.

 

Goal

Measure the total latency of a live stream over network from 10393 camera.
 

Setup

• Camera: NC393-F-CS
  • Resolution@fps: 1080p@30fps,  720p@60fps
  • Compression quality: 90%
  • Exposure time: 1.7 ms
  • Stream formats: mjpeg, rtsp
  • Sensor: MT9P001, 5MPx, 1/2.5″
  • Lens: Computar f=5mm, f/1.4, 1/2″
• PC: Shuttle box, i7, 16GB RAM, GeForce GTX 560 Ti
• Display: ASUS VS24A, 60Hz (=16.7ms), 5ms gtg
• OS: Kubuntu 16.04
• Network connection: 1Gbps, direct camera-PC via cable
• Applications:
  • gstreamer
  • chrome, firefox
  • mplayer
  • vlc
• Stopwatch: basic javascript

 

Notes table{ border-collapse: collapse; } td{ padding:0px 5px; border:1px solid black; } th{ padding:5px; border:1px solid black; background:rgba(220,220,220,0.5); }

Fig.2 Image sensor is rotated 90°[/caption] It's important to keep in mind that displays are refreshed progressively - so, if an image sensor is a rolling shutter type - their scan directions will coinside. -->
Table 1: Transfer times and data rate

Resolution/fps Image size1, KB Transfer time2, ms Data rate3, Mbps 720p/60 250 2 120 1080p/30 500 4 120

1 – average compressed (90%) image size
2 – time it takes to transfer a single image over network. Jitter is unknown. t = Image_size*1Gbps
3 – required bandwidth: rate = fps*Image_size

Camera output latency calculation

All numbers are for the given lens, sensor and camera setup and parameters. Briefly.

Sensor
Because of ERS each row’s latency is different. See tables 2 and 3.
 
Table 2: tROW and tTR

Resolution tROW1, us tTR2, us 720p 22.75 13.33 1080p 29.42 20 full res (2592×1936) 36.38 27

1 – row time, see datasheet. tROW = f(Width)
2 – time it takes to transfer a row over sensor cable, clock = 96MHz. tTR = Width/96MHz
 
Table 3: Average latency and the whole range.

Resolution tERS avg1, ms tERS whole range2, ms 720p 8 0.01-16 1080p 16 0.02-32

1 – average latency
2 – min – last row latency, max – 1st row latency

Exposure

tEXP < 1 ms – typical exposure time for outdoors. A display is bright enough to set 1.7 ms with the gains maxed.

Compressor

The compressor is implemented in fpga and works 3x times faster but needs a stripe of 20 rows in memory. Thus, the compressor will finish ~20/3*tROW after the whole image is read out.

tCMP = 20/3*tROW

Summary

tCAM = tERS + tEXP + tCMP

Since the image is read and compressed by fpga logic of the Zynq and this pipeline has been simulated we can be sure in these numbers.
 
Table 4: Average output latency + exposure

Resolution tCAM, ms 720p 9.9 1080p 17.9 Stopwatch accuracy

Not accurate. For simplicity, we will rely on the camera’s internal clock that time stamps every image, and take the javascript timer readings as unique labels, thus not caring what time they are showing.
 

Results

Fig.2 1080p 30fps

Fig.3 720p 60fps

 
GStreamer has shown the best results among the tested programs.
Since the camera fps is discrete the result is a multiple of 1/fps (see this article):

• 30 fps => 33.3 ms
• 60 fps => 16.7 ms

 

Resolution/fps Total Latency, ms Network+PC+SW latency, ms 720p@60fps 33.3-50 23.4-40.1 1080p@30fps 33.3-66.7 15.4-48.8

 

Possible improvements Camera

Currently, the driver waits for the interrupt from the compressor that indicates the image is fully compressed and ready for transfer. Meanwhile one does not have to wait for the whole image but start the transfer when the minimum of the compressed is data ready.

There are 3 more interrupts related to the image pipeline events. One of them is “compression started” – switching to it can reduce the output latency to (10+20/3)*tROW or 0.4 ms for 720p and 0.5 ms for 1080p.

Other hardware and software

In addition to the most obvious improvements:

• For wifi: use 5GHz over 2.4GHz – smaller jitter, non-overlapping channels
• Lower latency software: for mjpeg use gstreamer or vlc (takes an extra effort to setup) over chrome or firefox because they do extra buffering

Links

• Latency in live network video surveillance
• Wifi latencies, 2.4GHz & 5GHz
• This video compares different displays.
• About ERS

AndreyFilippov committed changes to the Elphel git project :
working on high disparity objects

Oleg Dzhimiev committed changes to the Elphel git project :
disabled, just in case

Oleg Dzhimiev committed changes to the Elphel git project :
check extension first

Oleg Dzhimiev committed changes to the Elphel git project :
set initial position and save to kml

Oleg Dzhimiev committed changes to the Elphel git project :
cache images