Model 333 camera is the third generation of the cameras designed at Elphel, Inc. The first one (model 303 in 2001) used the Axis ETRAX100LX Linux-optimized 32-bit 100MHz processor and a simple one-time programmable FPGA to interface the CMOS image sensor. That camera was designed for single-frame (not video) applications where the speed of software JPEG compression was not critical – one 1280x1024 frame required 5 seconds of CPU time.

The next Elphel camera (model 313 in 2003) targeted more general network cameras applications, trying to combine the high resolution images from CMOS sensors with the highest frame rate they were able to provide. At that time sensors were capable of 1280x1024 at 15 frames per second – 75 times faster than model 303 software-only compression could handle. To match that challenge I decided to use reconfigurable FPGA – technology that was new for me at that time. Before starting actual design I looked for existence of the proprietary solutions (“IP cores”) with similar specs to see if the goal is possible to achieve and how big/fast FPGA was needed.

It did not take long to find a good candidate – Xilinx Spartan-2e 300K gates chip (supported by free for download development tools from Xilinx) and soon I was ready to start FPGA hacking – my first complex FPGA design. And in three months there was a camera that could compress 15 frames per second at 1280x1024 resolution – more than any other network camera at that time. Flexibility of the reconfigurable FPGA was essential as many features were added and bugs fixed after the hardware was released, similar to how it is done with the software updates. The frame rate at the 1280x1024 resolution increased to 22 fps, support for higher resolution sensors (2, 3 and even 11 megapixels) was added when such sensors became available.

FPGA-based video processing in the model 313 camera demonstrated that such approach can significantly enhance performance of the software-based embedded systems – the camera with the total power consumption of just 3 watts has the performance of JPEG compression equivalent to that of 2.5-3.0 GHz PC-type computer. Having “hardware” speed such system retains flexibility of the software product.

When the model 313 design was mostly ready (mostly – as the FPGA/software code is never really finished) it turned out that 98% of the FPGA resources are used. Later new sensors became available that are faster than the model 313 JPEG encoder so it was a good time to think of a hardware upgrade.

High resolution combined with high frame rate already require a lot of processing power and earlier model 313 camera that served motion JPEG needed a PC with a 2.5GHz CPU to match it performance. And video (asymmetrical) compression at even higher frame rate would need a computational performance too high to be implemented in the camera by the universal processor – that make pure software solutions not practical.

Other possible solution could be a dedicated compressor IC. There are some available but most support only low resolution video (640x480 or less). Development of a new custom ASIC is expensive, time consuming and inevitably lags behind of the new compression algorithms. Such solution is not flexible, no upgrades or corrections of the errors are possible after the ICs are built.

As a result – there are no other network cameras capable of providing all three features (high resolution, high frame rate and low bit rate) simultaneously.

Most of the “hard job” of the video encoding in model 333 camera is done by a million-gate Xilinx Spartan 3 reconfigurable FPGA that supplements the general purpose processor that runs GNU/Linux and is responsible for the overall camera operation and streaming of the hardware-compressed video over the network.

Reconfigurable FPGA in the model 333 camera is able to outperform hypothetical 10GHz universal PC that will need at least hundred times more electrical power, it makes it possible to combine the “hardware” speed with flexibility of the software and allows usage of the new emerging video formats before they are finalized and constantly improve the system performance during the product lifetime.

Because of the similarity of the FPGA design to that of the software we apply highly productive FOSS development model to the hardware (FPGA) code development and release it with the free license (GNU/GPL). It gives our customers freedom to modify our products to better fit their specific requirements.

Some simple FPGAs are designed in a way similar to the schematic designs – with circuit diagrams where virtual components are used instead of real counters, registers, memories, and so on. This technique does not work well for complex designs and most of them look now very much similar to the software projects with the source files having preprocessor directives, modules, functions, variables declarations and operators.

There are two main hardware description languages in the industry: VHDL and Verilog HDL, and Verilog was chosen for Elphel designs.

The source code in Verilog will not look alien to the software developer, and if you open it for example with one of the KDE code editors – it will highlight the Verilog syntax.

So is FPGA programming the same as writing software? Ideally – probably yes, but it is still very different, especially for those who came to this frontier area from the software, not hardware side. As it was in earlier days of software design it is very important to understand what your code will be compiled into even when using high level directives.

There are at least two fundamental differences between FPGA and software code:

Propagation delays for the signals are analog and depend on multiple factors including distance between the data source and the receiver and number of inputs connected to one output. So accounting for delays and specifying timing constraints is an integral part of the overall project together with the the functional design. Because of the physical nature of the hardware, in some cases design can stop functioning properly after recompilation with different seed or modifications in unrelated modules – changes in mapping (assigning primitive cells to functional modules) and layout influence timing delays.

In the software different instructions are usually processed at different time, HDL code is compiled so that different operators are mapped to different parts of a chip and are usually active simultaneously.

On the top level parallelism is usually achieved by designing the system in such a way, that the data flow processing is split into conveyor-like small operations. In FPGA such pipe-type parallelism is usually more effective than separating the data flow into several sub-streams (i.e. splitting image into separate areas and processing each by different module instance), because it is easier to implement in hardware a single specific operation than a complex sequence of instructions as the processors driven by software do.

The data rate in each chain remains the same, so after the complex algorithm is split into sequence of simpler procedures it is possible to plan the required resources for each module and actually implement them. Some modules might be simple, some require manual optimizations and careful planning of the data dependence and assigning register operations to particular time slots before the coding can start. A good example of such module could be an 8-point discrete cosine transform (DCT). Chain of two of such modules (with a memory buffer between them) is capable of a 2-d 8x8 DCT - one of the most computationally intensive parts of baseline JPEG, MPEG and Ogg Theora codecs.

Modern FPGAs are mature devices that have many features common to different families and manufacturers. They have:

Similar to its predecessor model 333 camera circuitry (Figure 1) is designed to be flexible and allow future modifications (i.e. new sensor support) without any changes on the board. It consists of a universal computer with Axis Communications ETRAX100LX 32bit Linux-optimized processor with embedded Ethernet MAC and other ports, 32MB of system SDRAM and 16MB flash. FPGA plays central role in video processing and is connected to the rest of the system using 32bit wide system bus, providing both PIO an DMA access. Some of the FPGA I/O pins are connected directly to the connector that is used to interface interchangeable sensor boards. All these pins are reprogrammable and it is easy to change the pins designation to match control signals for different sensors including those that were not available when the camera board was released.

Figure 1Model 333 Camera Block Diagram

Programmable clock generator adds to flexibility of the design – all of the system clock frequencies are adjustable by the software.

Maximal data transfer rate even in the DMA mode (80Mbytes/sec) is still not enough to provide a scratch pad memory access for video compression and the on-chip FPGA memory is far less than needed for the high resolution images (it is only 54 Kbytes in a million-gate Spartan-3). The fast access to large memory led to addition of the dedicated 32MB DDR SDRAM connected to the FPGA to supplement its computational resources with large scratch pad memory.

FPGA implementation of the Ogg Theora encoder is based on the format specifications written by Xiph.org and currently available at http :// www.theora.org /doc/Theora_I_ spec.pdf. The code in Elphel model 333 camera is primarily targeted to the applications where the camera does not move, and so motion compensation is not implemented yet. The other shortcut – omission of the loop filter that improves visual image quality at high compression ratios (low quality) by making borders between 8x8 pixel blocks less visible. The camera uses reconfigurable FPGA, so it will be possible to add this functionality in the future code releases.

The first step was to adjust the overall algorithm to the hardware implementation and present it as a chain of the processing modules (Figure 2).Special attention was needed to organize data transfer to and from external memory because, in contrast to the highly parallel distributed internal resources of the FPGA, that memory is a single physical device with a limited I/O bandwidth.

Figure 2FPGA Code Diagram

Modern FPGAs are powerful enough to handle such complex tasks as a real time video compression – traditional application area of custom ASICs. FPGA implementation of Theora video encoder runs at 30fps for 1280x1024 resolution, 12fps for full 2048x1536 and several hundred frames per second for smaller windows – that makes Elphel model 333 the first high resolution network camera to support true video compression.

All the FPGA code (Verilog source, constraints and project settings) of Elphel products comes with GNU/GPL, and FPGA development does not require expensive tools – Xilinx provides free for download software. These tools can be used to compile the code into actual image that can be loaded to the on-board FPGA, so software developer can use the camera (or similar products based on the reconfigurable devices) to experiment with the FPGA design.

Theora video codec is a new one and is not yet as popular as traditional MPEG, but it can be freely used and modified without any royalty payments. That makes it the only high performance video codec suitable for inclusion into most GNU/Linux distributives and it is already supported by many of the available video players.

We hope that our products will help to introduce FPGA design to software developers and will contribute to the acceptance of Ogg Theora in the new application area of network cameras.