Fig.1 Live stream latency testing

Recently we had an inquiry whether our cameras are capable of streaming low latency video. The short answer is yes, the camera’s average output latency for 1080p at 30 fps is ~16 ms. It is possible to reduce it to almost 0.5 ms with a few changes to the driver.

However the total latency of the system, from capturing to displaying, includes delays caused by network, pc, software and display.

In the results of the experiment (similar to this one) these delays contribute the most (around 40-50 ms) to the stream latency – at least, for the given equipment.

 

Goal

Measure the total latency of a live stream over network from 10393 camera.
 

Setup

• Camera: NC393-F-CS
  • Resolution@fps: 1080p@30fps,  720p@60fps
  • Compression quality: 90%
  • Exposure time: 1.7 ms
  • Stream formats: mjpeg, rtsp
  • Sensor: MT9P001, 5MPx, 1/2.5″
  • Lens: Computar f=5mm, f/1.4, 1/2″
• PC: Shuttle box, i7, 16GB RAM, GeForce GTX 560 Ti
• Display: ASUS VS24A, 60Hz (=16.7ms), 5ms gtg
• OS: Kubuntu 16.04
• Network connection: 1Gbps, direct camera-PC via cable
• Applications:
  • gstreamer
  • chrome, firefox
  • mplayer
  • vlc
• Stopwatch: basic javascript

 

Notes table{ border-collapse: collapse; } td{ padding:0px 5px; border:1px solid black; } th{ padding:5px; border:1px solid black; background:rgba(220,220,220,0.5); }

Fig.2 Image sensor is rotated 90°[/caption] It's important to keep in mind that displays are refreshed progressively - so, if an image sensor is a rolling shutter type - their scan directions will coinside. -->
Table 1: Transfer times and data rate

Resolution/fps Image size1, KB Transfer time2, ms Data rate3, Mbps 720p/60 250 2 120 1080p/30 500 4 120

1 – average compressed (90%) image size
2 – time it takes to transfer a single image over network. Jitter is unknown. t = Image_size*1Gbps
3 – required bandwidth: rate = fps*Image_size

Camera output latency calculation

All numbers are for the given lens, sensor and camera setup and parameters. Briefly.

Sensor
Because of ERS each row’s latency is different. See tables 2 and 3.
 
Table 2: tROW and tTR

Resolution tROW1, us tTR2, us 720p 22.75 13.33 1080p 29.42 20 full res (2592×1936) 36.38 27

1 – row time, see datasheet. tROW = f(Width)
2 – time it takes to transfer a row over sensor cable, clock = 96MHz. tTR = Width/96MHz
 
Table 3: Average latency and the whole range.

Resolution tERS avg1, ms tERS whole range2, ms 720p 8 0.01-16 1080p 16 0.02-32

1 – average latency
2 – min – last row latency, max – 1st row latency

Exposure

tEXP < 1 ms – typical exposure time for outdoors. A display is bright enough to set 1.7 ms with the gains maxed.

Compressor

The compressor is implemented in fpga and works 3x times faster but needs a stripe of 20 rows in memory. Thus, the compressor will finish ~20/3*tROW after the whole image is read out.

tCMP = 20/3*tROW

Summary

tCAM = tERS + tEXP + tCMP

Since the image is read and compressed by fpga logic of the Zynq and this pipeline has been simulated we can be sure in these numbers.
 
Table 4: Average output latency + exposure

Resolution tCAM, ms 720p 9.9 1080p 17.9 Stopwatch accuracy

Not accurate. For simplicity, we will rely on the camera’s internal clock that time stamps every image, and take the javascript timer readings as unique labels, thus not caring what time they are showing.
 

Results

Fig.2 1080p 30fps

Fig.3 720p 60fps

 
GStreamer has shown the best results among the tested programs.
Since the camera fps is discrete the result is a multiple of 1/fps (see this article):

• 30 fps => 33.3 ms
• 60 fps => 16.7 ms

 

Resolution/fps Total Latency, ms Network+PC+SW latency, ms 720p@60fps 33.3-50 23.4-40.1 1080p@30fps 33.3-66.7 15.4-48.8

 

Possible improvements Camera

Currently, the driver waits for the interrupt from the compressor that indicates the image is fully compressed and ready for transfer. Meanwhile one does not have to wait for the whole image but start the transfer when the minimum of the compressed is data ready.

There are 3 more interrupts related to the image pipeline events. One of them is “compression started” – switching to it can reduce the output latency to (10+20/3)*tROW or 0.4 ms for 720p and 0.5 ms for 1080p.

Other hardware and software

In addition to the most obvious improvements:

• For wifi: use 5GHz over 2.4GHz – smaller jitter, non-overlapping channels
• Lower latency software: for mjpeg use gstreamer or vlc (takes an extra effort to setup) over chrome or firefox because they do extra buffering

Links

• Latency in live network video surveillance
• Wifi latencies, 2.4GHz & 5GHz
• This video compares different displays.
• About ERS

AndreyFilippov committed changes to the Elphel git project :
working on high disparity objects

Oleg Dzhimiev committed changes to the Elphel git project :
disabled, just in case

Oleg Dzhimiev committed changes to the Elphel git project :
check extension first

Oleg Dzhimiev committed changes to the Elphel git project :
set initial position and save to kml

Oleg Dzhimiev committed changes to the Elphel git project :
cache images

Oleg Dzhimiev committed changes to the Elphel git project :
1.&edit - edit mode = set initial location and azimuth on the leaflet map 2. &experimental - experimental mode

Oleg Dzhimiev committed changes to the Elphel git project :
standalone loop player for source files

Oleg Dzhimiev committed changes to the Elphel git project :
fixed reset view

‎Change the default ip address

← Older revision Revision as of 18:36, 5 July 2017 (One intermediate revision by the same user not shown)Line 272: Line 272:     ==<font color="blue">Other info</font>== ==<font color="blue">Other info</font>==  +====Change default ip address====  +* '''nano''' or '''vi'''  + ssh root@192.168.0.9  + ''pass''  + root@elphel393:~# nano /etc/elphel393/init_elphel393.py  + ''edit - save''  +  +'''or'''  +* '''sed'''  + root@elphel393:~# sed -i.bkp 's/192.168.0.9/10.xx.xx.xx/' /etc/elphel393/init_elphel393.py  + root@elphel393:~# sed -i.bkp 's/255.255.255.0/xxx.xxx.xxx.xxx/' /etc/elphel393/init_elphel393.py  +  +Then:  + #boot from flash?  + root@elphel393:~# overlay_sync 1  + root@elphel393:~# shutdown -hP now  + #powercycle  +  + #boot from card?  + root@elphel393:~# sync  + root@elphel393:~# reboot -f  +  ====Set up histogram window and autoexposure parameters==== ====Set up histogram window and autoexposure parameters====  [[Autoexposure|Read article]] [[Autoexposure|Read article]] Oleg

‎Change the default ip address

← Older revision Revision as of 18:36, 5 July 2017 (6 intermediate revisions by the same user not shown)Line 124: Line 124:    http://192.168.0.9:2325/mimg   http://192.168.0.9:2325/mimg    http://192.168.0.9:2326/mimg   http://192.168.0.9:2326/mimg  +  +====RTSP====  +* turn on:  + ~root@elphel393:~# /usr/bin/str  +* url:  + rtsp://192.168.0.9:554  + rtsp://192.168.0.9:556  + rtsp://192.168.0.9:558  + rtsp://192.168.0.9:560     ====GStreamer==== ====GStreamer==== Line 130: Line 139:  </font> </font>  * mjpeg and rtsp * mjpeg and rtsp −* playback latencies (gstreamer)  − 1080p - ~70ms   * More examples at [[Using_gstreamer#Display|Using GStreamer]] * More examples at [[Using_gstreamer#Display|Using GStreamer]]    Line 260: Line 267:     ==<font color="blue">Firmware images</font>== ==<font color="blue">Firmware images</font>==  +* [http://community.elphel.com/files/393/20170627/ '''20170627''']  * [http://community.elphel.com/files/393/20170623/ 20170623] * [http://community.elphel.com/files/393/20170623/ 20170623]  * [http://community.elphel.com/files/393/20170531/ 20170531] * [http://community.elphel.com/files/393/20170531/ 20170531] −* [http://community.elphel.com/files/393/20170124/ 20170124]      ==<font color="blue">Other info</font>== ==<font color="blue">Other info</font>==  +====Change default ip address====  +* '''nano''' or '''vi'''  + ssh root@192.168.0.9  + ''pass''  + root@elphel393:~# nano /etc/elphel393/init_elphel393.py  + ''edit - save''  +  +'''or'''  +* '''sed'''  + root@elphel393:~# sed -i.bkp 's/192.168.0.9/10.xx.xx.xx/' /etc/elphel393/init_elphel393.py  + root@elphel393:~# sed -i.bkp 's/255.255.255.0/xxx.xxx.xxx.xxx/' /etc/elphel393/init_elphel393.py  +  +Then:  + #boot from flash?  + root@elphel393:~# overlay_sync 1  + root@elphel393:~# shutdown -hP now  + #powercycle  +  + #boot from card?  + root@elphel393:~# sync  + root@elphel393:~# reboot -f  +  ====Set up histogram window and autoexposure parameters==== ====Set up histogram window and autoexposure parameters====  [[Autoexposure|Read article]] [[Autoexposure|Read article]] Oleg

AndreyFilippov committed changes to the Elphel git project :
fixed lazy eye correction

Oleg Dzhimiev committed changes to the Elphel git project :
hide dirs: .xxx and _xxx, 'showall' to reveal

AndreyFilippov committed changes to the Elphel git project :
more on aligning

AndreyFilippov committed changes to the Elphel git project :
working on alignment correction

AndreyFilippov committed changes to the Elphel git project :
minor disparity correction for infinity for disturbed camera

AndreyFilippov committed changes to the Elphel git project :
correcting extrinsics after being changed

Oleg Dzhimiev committed changes to the Elphel git project :
removed trailing spaces

Oleg Dzhimiev committed changes to the Elphel git project :
Merge branch 'gulp'

Oleg K Dzhimiev committed changes to the Elphel git project :
Merge branch 'master' into 'gulp'