Elphel Develoment Blog

1. Get X Virtual FrameBuffer [1] sudo apt-get install xvfb 2. Launch ImageJ ("cd" to the ij.jar directory): Xvfb :15 & DISPLAY=:15 java -Xmx12288m -jar ij.jar -run "TestIJ Plugin" Comments: TestIJ Plugin is the name of the compiled plugin in the ImageJ menu. No need to specify a subfolder. :15 is an example. Links that helped: Source 1 [2] Source 2 [3] Source 3 [4] [1] http://en.wikipedia.org/wiki/Xvfb [2] http://stackoverflow.com/questions/4599618/how-to-add-3rd-party-jars-to-compile-time-classpath-in-jgrasp [3] http://imagej.1557.n6.nabble.com/How-to-set-DISPLAY-environment-variable-when-running-ImageJ-with-NO-SHOW-option-td3686181.html [4] http://cmci.embl.de/documents/100922imagej_cluster

Seeing the impressive images of the Elphel-Eyesis 4pi camera I thought it’s time to tell you about the HomeSide 720°. Like the Eyesis its purpose is to capture panorama frames with a framerate of 5fps. The major difference is that the HomeSide 720° is mounted on a helmet. To have an acceptable weight it consists of only two instead of eight Elphel 353 delivering one forth of the resolution the 4pi does. Thus the camera is able to record 30MPix frames before stitching. Additionally it’s reconfigureable to enable HDR panorama frames. More interesting probably is the purpose it was built for. We created the assembly for indoor virtual tours. After several drawbacks we finally have an approach which works very well. We do auto leveling, auto stabilization and path extraction by image analysis only. Furthermore we recognize crossing points where the user can decide where to go when the tour is shown in the player. This is not so easy since we neither have GPS nor IMU data. Nevertheless its possible. All this information goes into our new webplayer which reassembles the images to a virtual tour. Have a look at the HomeSide 720° Virtual Tour [1] Click into the player and use the cursor keys to navigate. You may also click and drag to change the point of view.  This tour was recorded with 10MPix i.e. one Elphel 353 with two sensors. Important: The pi symbols shows a rendered tour, not recorded by the camera At the moment we are improving the image quality. We are also looking for a partner to drive the development even faster to create stunning indoor virtual tours. [1] https://homeside.at/html/inside-demo.html

[1] It has been a long while since my last blog entry [2] in regards to river view panoramas. In the meantime the recording setup runs basically stable (putting aside minor problems with loose connectors) even under rough conditions (see also the gallery "Making Of [3]" at the end of this post). I just came back from artist-in-residency stays in Varanasi/Benares [4] and Guwahati [5] in India, that enabled me to have a few extensive recording sessions on various vessels like house boats, motor and rowing boats on Ganges River [6] - for one the most sacred river to Hindus and probably most worshipped river on the planet, next to being one of the most polluted rivers [7] of the world - and Brahmaputra [8] River in Assam. Many thanks go to Kriti Gallery [9] in Varanasi and the Periferry [10] project in Guwahati for hosting me and helping me to get onboard. Web Player Meanwhile I also just finished the "River View Web Player" as a public online Beta Version. It runs on openlayers [11] and geodjango [12] and features the growing archive and collection of river views (up to now including views of Ganges, Brahmaputra, Danube and Nile River). There are still plenty of things to polish up, some image material to be retouched and/or uploaded, but see yourself here [13]: [caption id="attachment_3805" align="alignnone" width="320" caption="River View Web Player"] [14][/caption] Using Elphel I am using an Elphel 353 equipped with 8 or 16mm movie lenses from the seventies to acquire this imagery. The camera delivers a Window-of-Interest video stream of 2592 x 48 pixel size with a variable frame rate between 1 and 300fps (changed on the fly according to the speed of the vessel and distance to the object of focus. That is done manually and under visual control: similar to the auto-focus mechanism, it is also a question of choice and decision and therefore not easily automatable). The video stream is then processed and saved by custom software. I am not using Elphel's internal linescan/photofinish for now, since it still behaves pretty unstable and unpredictable when changing parameters like TRIGGER, VIRT_KEEP and VIRT_HEIGHT on the fly. Also, having a line per frame allows me for a more fine-graded control and preview as long as enough frames are delivered. Elphel's internal linescan mode, however, works quite well and stable if it runs as fast as it goes or once you have fixed a setup - as some testing on Austrian Highways prove (find more here [15]): [caption id="attachment_3803" align="alignnone" width="700" caption="Elphel 353 in Linescan mode on Austrian Highway A1 - approx. 120km/h, 2000 lps"] [16][/caption] Sources The source code of both, my recording software malisca [17] and the web app and player [18], are open on my github repository. Making Of / Gallery A brief guided tour to Ganges and Brahmaputra river recordings in pictures: [caption id="attachment_3859" align="alignnone" width="500" caption="On Ganges"] [19][/caption] [caption id="attachment_3865" align="alignnone" width="640" caption="Elphel on Ganges River"] [20] [21][/caption] [caption id="attachment_3857" align="alignnone" width="500" caption="river recording setup: Thinkpad, Elphel 353, battery pack and USB-GPS-receiver"] [22][/caption] [caption id="attachment_3860" align="alignnone" width="500" caption="another shot of recording setup (including an improvised tent)"] [23][/caption] [caption id="attachment_3862" align="alignnone" width="500" caption=".. staring into the lens .."] [24][/caption] [caption id="attachment_3861" align="alignnone" width="500" caption=".. obstacles (pontoon bridge) .."] [25][/caption] [caption id="attachment_3863" align="alignnone" width="500" caption="lunch break on Ganges River"] [26][/caption] [caption id="attachment_3864" align="alignnone" width="500" caption="Elphel on Brahmaputra River"] [27][/caption] [caption id="attachment_3867" align="alignnone" width="375" caption="Not yet, but I would love this setup to be powered by solar energy ...."] [28][/caption] [1] http://blog.elphel.com/wp-content/uploads/2012/02/0900.jpg [2] http://blog.elphel.com/2010/11/elphel-as-a-line-scan-camera-for-river-panoramas/ [3] http://blog.elphel.com#making_of [4] https://en.wikipedia.org/wiki/Varanasi [5] https://en.wikipedia.org/wiki/Guwahati [6] https://en.wikipedia.org/wiki/Ganges [7] http://articles.timesofindia.indiatimes.com/2007-03-20/india/27875507_1_ganga-river-wwf [8] https://en.wikipedia.org/wiki/Brahmaputra_River [9] http://www.kritigallery.com/ [10] https://periferry.wordpress.com/about/ [11] http://openlayers.org/ [12] http://geodjango.org/ [13] http://play.riverstudies.org [14] http://play.riverstudies.org [15] http://www.riverstudies.org/testing/2011-04-27--westautobahn-II/ [16] http://www.riverstudies.org/testing/2011-04-27--westautobahn-II/ [17] https://github.com/backface/malisca [18] https://github.com/backface/riverstudies-django-app [19] http://blog.elphel.com/wp-content/uploads/2012/02/0500.jpg [20] http://blog.elphel.com/wp-content/uploads/2012/02/1000.jpg [21] http://blog.elphel.com/wp-content/uploads/2012/02/11001.jpg [22] http://blog.elphel.com/wp-content/uploads/2012/02/0600.jpg [23] http://blog.elphel.com/wp-content/uploads/2012/02/0700.jpg [24] http://blog.elphel.com/wp-content/uploads/2012/02/09001.jpg [25] http://blog.elphel.com/wp-content/uploads/2012/02/0800.jpg [26] http://blog.elphel.com/wp-content/uploads/2012/02/0910.jpg [27] http://blog.elphel.com/wp-content/uploads/2012/02/0920.jpg [28] http://blog.elphel.com/wp-content/uploads/2012/02/12001.jpg

Motivation While working on the second generation of the Eyesis panoramic cameras, we decided to try go from capturing the series of the individual panoramic images to the 3d reconstruction. There are multiple successful implementations of such process, we just plan to achieve higher precision of capturing the 3d worlds using Elphel ability to design and build the hardware specific for such purpose. While most projects are designed to work with the standard off-the-shelf cameras, we are working on building the cameras together with the devices and methods for these cameras calibration.  In order to be able to precisely determine the 3-d locations of the features registered with the cameras we plan first go as far as possible to precisely map each pixel of each sub-camera (of the composite camera) image to the ray in space. That would require at least two distinctive steps: map each pixel of each sub-image to the ray in the composite camera frame system and measure the composite camera egomotion to find the absolute pixels-to-rays mapping. The first step can be done by calibration of the camera in the laboratory, measuring the position and orientation of each sub-camera relative to the composite camera frame and simultaneously measuring camera lens distortions. To implement the second step we plan to combine the inertial data from the high resolution/high bandwidth inertial measurement unit (IMU) with the optical data from the camera itself. This post is dedicated to the first part, the static pixel mapping of the composite camera. The complete camera has 26 sub-camera modules and we are building a goniometer type camera calibration machine that will rotate the finished camera around two axes exposing the large calibration pattern (3.0m by 2.6m) to different areas of each of the sub-camera sensors. Registered images will be also used to correct each lens aberration as described in the earlier blog posts [1] Registering the pattern grid We first designed the grid pattern [2] for measuring (and correcting) the lens aberrations - it has less aliasing artifacts and more uniform spectrum than plain checkerboard pattern normally used for distortion measurement while still being easy to detect and register. Processing starts from the Bayer mosaic image acquired from the camera and for distortion measurements only the green channel is used. Figure 1a shows the grid that occupies most of the image sensor area, other images may have the pattern in smaller areas, so the pattern registration starts with detecting pattern in at least some areas. For that the smaller (normally 128x128 pixels) square areas of green pixels are extracted as shown on Fig.1b the image is rotated 45 degrees from the original on fig.1a because green channel uses half of the sensor pixels, located in say "black" cells of the checker board pixel grid. [caption id="attachment_3565" align="alignleft" width="537" caption="Fig.1 Pattern grid registration: a) registered image. b) 128pix*128pix region for green channels. c) power spectrum (gamma=0.2). d) clusters around first maximums in 2 directions. e) 64x64 region centered around the estimated grid node. f) simulated grid with window function applied. g) mixed cross/phase correlation of registered and simulated arrays. h) combination of correlations with 4 half-pixel shifted models. i) result pixel X coordinate for each of the registered grid nodes. j) same for the pixel Y coordinate"] [3][/caption] Next step is to calculate the spectrum of this pattern - Fig. 1c shows the power spectrum to the power of 0.2 to somewhat equalize the harmonics amplitude (it worked better than logarithm function). Pixel clusters around the first several maximums along the two pattern axes were extracted and used for initial measurement of the two pattern translation vectors (Fig. 1d).  If no pattern is detected the scanning continues, instead of the sequential scanning I used reverse-bit order to reduce number of trials before hitting the area with the pattern. After the program  detects the first area with the pattern it continues with the more precise measurement of the correlation between the acquired image and the simulated pattern,  calculated for the same grid translation vectors. Fig. 1e shows the 64x64 pixels fragment of the registered image centered around the estimated grid node (before multiplying by a window function) and Fig. 1f shows the matching simulated one with the window applied. Correlation is performed through direct and reverse Fourier transforms, run-time parameter allows changing from the cross-correlation to the phase correlation. Fig. 1g shows result of such operation applied to images on Fig. 1e and 1f. While there is only one copy of the acquired image around the grid node shown on Fig. 1e, it is possible to generate the simulated image (Fig. 1f) with different sub-pixel shifts, combining the results of their correlation with the same source image can reduce effects of the aliasing and increase the resolution of the correlation result. The next image (Fig. 1h) shows the result of combining 4 of 64x64 pixels correlations where the simulated images are generated with 0 and 0.5 pixel shifts in each of the two directions, using the center 64x64 of the combined 128x128 image and application of a low-pass filter. This data is then used to calculate the coordinates of the grid node with one of the several methods - fitting the quadratic function or finding the centroid of the top of the maximum, cut at a specified threshold or area. The whole process of calculating the locations of the grid nodes is ran twice. On the first pass program starts from the first node detected and uses the grid parameters to estimate the locations of the neighbor nodes and use this estimation to generate the local pattern simulation used for correlation. This process generates a wave from the initially detected point that propagates until all the connected grid nodes are processed and the correlation contrast (calculated as the ratio of the average signal inside the center circle and that in a ring that falls between the expected maximums) falls below the specified threshold. The program uses quadratic approximation for the simulated pattern generation to compensate for the local lens distortions - while not critical for the lens used in this work, that was important to fit the grid to the fisheye lens used in the first Eyesis camera. The second pass refines the location of the grid nodes detected in the first pass - this time the locations of the neighbors already known are used for the local pattern grid generation. The result of this procedure is the array of grid points organized by the grid coordinates (u,v) with half of them (even sum of u+v) being "positive" and the other half (odd sum of u+v) being inverted. This data is stored as a multi-slice TIFF image, two of the slices: sensor pixel X coordinate  and sensor pixel Y are shown on the Fig. 1i and Fig. 1j, respectively. [caption id="attachment_3538" align="alignleft" width="230" caption="Fig.2a Grid curvature - distance in pixels from the registered pattern grid node location and the average of it 8 neighbors (full scale 0.2 pixels). Visible center seam of the two pattern panels."] [4][/caption] Evaluating results of the pattern registration When several methods of pattern grid registration were implemented I needed some way to estimate the precision of the results, to be able to compare the influence of the different variable parameters (correlation size, window function, cross/phase correlation ratio) on this "registration quality". Most interesting for us was how repetitive the registration results are so how their invariants would differ for the images acquired in at least approximately the same conditions. [caption id="attachment_3539" align="alignright" width="270" caption="Fig. 2c Grid curvature - distance in pixels from the registered pattern grid node location and the average of it 8 neighbors. One half (corresponding to top of the Fig.2a) is shown."] [5][/caption] [caption id="attachment_3683" align="alignleft" width="230" caption="Fig. 2b Grid curvature calculated with equalization of the pattern brightness"] [6][/caption] I used the following intra-frame estimate of the registration errors. First the program calculated the grid curvature as a difference between the selected grid node location (pixel X and Y values) and the average value calculated from the node neighbors. This value (shown on Fig. 2a-c) is dependent on the actual curvature of the registered grid (caused by the lens distortion), the errors of the physical pattern attached to the wall and the variations of the results of the registration method. All of these three contributing factors reveal themselves on these images. The circular structures are related the optical distortions, the bright column in the middle of the Fig. 2a is caused by the small misalignment of the 2 halves of the physical pattern - it consists of the two 1.5m by 2.6m plastic panels attached to the wall. Each of the panels maybe (and actually is as we'll see later) slightly distorted from the perfectly rectangular shape, and there is some error in alignment of the panels to each other. [caption id="attachment_3691" align="alignright" width="324" caption="Fig. 2d Pattern grid flat field correction"] [7][/caption] There is also some "noise" visible on Fig. 2a (has to be smoothed out for 3d visualization on Fig.2c), and checker-board artifacts mostly visible near the corners of the images. These artifacts are likely caused by vignetting which has more effect on white parts of the pattern and so biases the registered location when the the overall brightness gradient coincides  with the direction connecting the two opposite white quadrants of the pattern around the node.  Vignetting correction that we'll incorporate in the same measurement cycle will reduce this effect and improve the pattern grid node registration. Update: yes, these artifacts were in fact caused by the lens vignetting and uneven illumination of the pattern. When the pattern images were equalized (see Fig. 2d) the artifacts were successfully eliminated (Fig. 2b). At this stage it is insignificant what causes the brightness gradients - lens or the light source, and the "flat-field" correction is only needed for the areas of the image where the test pattern is detected. The equalization is implemented after the first pass of of measuring the grid node coordinates. The program calculates average value of the pixels around these already known coordinates using the same window function as for the correlation. This method seems to work as is, but it can be improved to probe average pixel values around the estimated centers of white and black cells, in that case the result will less depend on the initial precision so it will work for lenses with higher distortions (as the fisheye ones). When the average intensity is calculated for every known grid node, the border node values are removed because their value is influenced by the pixels outside of the pattern area. Next the average intensity map is expanded by extrapolating the measured intensity data outwards from the known cells to estimate the average intensity values near the pattern edges and beyond, so the low-pass filter would not skew the result. The average intensity map is then applied to the original image before refining the pattern grid node locations during the second pass (black area on Fig. 2d is not a privacy filter, it is just outside of extrapolation area ). [caption id="attachment_3537" align="alignleft" width="230" caption="Fig.3a Registration errors as distances from the registered grid nodes to the calculated from the neighbors, assuming constant curvature. Full scale is 0.2 pixels"] [8][/caption] The high frequency variations on the Fig. 2a are caused by the registration errors - measurements described later did not show any detectable node-to-node differences on the pattern, only the gradual changes. To separate gradual variations caused by the lens and pattern distortions I subtracted the estimated (from the neighbor nodes) grid node location from the registered one, assuming the constant local curvature, the results are shown on Fig. 3a and 3b. [caption id="attachment_3540" align="alignright" width="307" caption="Fig.3b Registration errors as distances from the registered grid nodes to the calculated from the neighbors, assuming constant curvature."] [9][/caption] The pattern discontinuity in the center is still present, but in the other areas the effects of the gradual variations are compensated. The checkerboard artifacts visible the Fig. 2a are hidden on Fig 3a, because the variations from the estimated (from the constant curvature) locations in these areas are opposite in direction, but same in length, so these artifacts just add to the average error value in the corners. The root mean square of these differences was used to compare different registration methods. We plan to make more measurements of the registration errors with the complete multi-channel cameras when the precise rotation machine will be built, currently I used just a single camera mounted on a tripod. Absolute mapping of the pattern grid with the laser pointers The pattern grid registration procedure described above resulted in arrays of the pixel coordinates of the pattern grid nodes,but there was no information about the absolute position of the measured pattern cells relative to the pattern as a whole. In some cases when the corner of the whole pattern is visible and the approximate camera orientation is known (so one of the 4 corners can be identified) it is possible to make absolute mapping, but we decided to supplement the pattern with the four laser diodes pointed to the known cells in the four quadrants of the pattern. With the lasers controlled by the software it was easy to detect location of the laser spots by subtracting a pair of images (with and without the laser spot). The precision of the laser spot location is not critical as long as those spots safely identify known white cells of the pattern grid. Absolute mapping of the pattern grid nodes is needed for both finding the extrinsic parameters of each sub-camera (position and orientation relative to the composite camera frame) and accounting for the non-ideal pattern grid when measuring the camera lens distortions. Composite camera and the camera rotation machine combined model Starting calibration of the composite camera I created a model to describe the location and orientation of each of the camera 26 sub-cameras relative to the composite camera frame, rotation of the composite camera in the calibration machine we are building (fixed horizontal rotation axis and moving camera vertical axis), deviations of the machine (distance between the axes,  difference from the 90 degree angle between the axes, two angles to determine actual orientation of the horizontal rotation axis) and location of the machine reference point (point on the horizontal rotation axis closest to the camera rotation axis) relative to the frame attached to the target. Each of the sub-cameras is described by a set of 7 intrinsic parameters based on a radial distortion model (focal length, pixel coordinates of the lens axis and 4 coefficients describing distortion polynomial) and a pixel-mapped array to represent the difference from the radial distortion model. The pattern grid is represented as a two-dimensional array (U,V) of the 3-d points. For the first experiments where all the images were acquired from approximately the same point the grid was considered flat (z=0), but the provision was made to store the full 3-d data for each pattern grid node to account for non-flatness of the physical pattern. The composite camera is designed to capture all images at the same time and each of the result images is uniquely identified by a timestamp (common to all sub-cameras) and the sub-camera number. While only the sub-camera intrinsic parameters (7 with the selected radial distortion)  and the 6 extrinsic ( 3 rotations and 3 translation) are sufficient to project each of the pattern grid points to the sensors I used a complete camera kinematic model to be able to impose constraints for the groups of images, such as that all the relative positions/orientations of the sub-cameras relative to the composite camera are common, all the parameters describing the calibration machine but the 2 variable rotations are also common, and these two variable angles are common for the images acquired from the different sub-cameras that share the same timestamp. Such model makes each image dependent on 15 extrinsic parameters and the conversion from 15 to 6 is common for each pattern node projection in the image. [caption id="attachment_3567" align="alignleft" width="486" caption="Fig.4 Images registered from the same camera position (9 of 15 shown)"] [10][/caption] The fitting of the parameters of the model to the series of pattern grid images is based on Levenberg-Marquardt algorithm (LMA)  and the deviations to minimize are differences in X and Y directions between the calculated locations of the pattern nodes projections on the sensor and their registered locations. So each grid node on each registered image being processed constitutes two separate samples for the algorithm. This algorithm involves calculating Jacobian matrix of the partial derivatives. Most of the partial derivatives with respect to the minimal set of 7+6=13 parameters are different for each pattern node of each image, we do not need to calculate them for each of the 22 parameters because conversion from 22 to 13 (both values and partial derivatives) can be calculated once per image and then the Jacobian calculation for 22 parameters can be done by multiplying Jacobian for 13 parameters by the 22x13 matrix. The implementation of the LMA is designed to have sequences of programmable strategies, each specifying a subset of the grid images to process, set of parameters to adjust and if they are individual for each image, shared by all or by the groups of the images. Program can run strategies automatically or stop after each iteration step or one strategy series. [caption id="attachment_3542" align="alignright" width="253" caption="Fig. 5a Reprojection error of the pattern grid after fitting camera extrinsic parameters and the intrinsic ones assuming polynomial radial distortions, shown separately for X and Y directions. Full scale is +/- 2mm"] [11][/caption] The experiment used a set of 15 images with a single camera mounted on a tripod at 2.3m from the pattern and rotated around approximately vertical axis to five positions in the range of ±30° for three different tilts  in the range of ±15°. Nine of the 15 such images are shown on Fig. 4. The actual acquired images were already converted to the grid arrays and stored as multi-slice TIFF images as described above, so each new run of the program did not require re-running the pattern registration process. In addition to the two slices shown on Fig. 1e and 1f (pixel X and pixel Y coordinates) those files had additional slices of integer values U and V representing absolute grid coordinates of the same pattern nodes available after the absolute mapping with laser pointers. Measuring the pattern geometry Radial distortion model provides rather accurate projection of the pattern grid nodes to the image sensor pixel coordinates so I expected the physical pattern distortions (non-flatness and stretching of the material) to be the next source of mismatch between the model and the registered image, we've already seen them in the registration process as illustrated on Fig.2 and 3. And after I ran the LMA on a set of 15 images and it converged on parameter vector, I mapped the reprojection errors (differences between calculated and measured pixel coordinates) to the grid coordinates. The nine images of Fig. 5a (animated GIF) and Fig. 5b (color) match the acquired images shown on Fig.4, i.e. top left (of nine) image shows the top left portion of the pattern. [caption id="attachment_3541" align="alignright" width="252" caption="Fig. 5b Reprojection pattern grid errors in pseudo-colors. Different colors indicate different directions, brightness - the absolute value of the difference."] [12][/caption] Each image frame on Fig. 5a and 5b represents the minimal straight rectangle that includes the full pattern, each sub-image covers different but overlapping part of the whole pattern. The rectangular physical pattern itself with the sides vertical and horizontal is represented here as tilted, because it is tilted in pattern UV coordinates as the pattern rows and columns are tilted by 5° to reduce aliasing artifacts. Animated images on Fig. 5a alternate between vertical and horizontal deviations of the measured grid node point coordinates and the ideally periodic ones used when the LMA was initially ran, color image Fig. 5b combines the vector components into a single image where direction is represented by the color and the absolute value - by the intensity. These partial images demonstrated a good match in the overlapping areas and the full set of images was acquired with a good overlapping coverage of the full pattern area. So the next step was to calculate the smooth masks to reduce the influence of the sensor-cropped grids and combine the individual images into the one covering the full pattern. The individual images multiplied by the weight function of the camera mapped to the pattern coordinates were added together and then per-pixel divided by the sum of those mapped weight functions. The result was close to the images shown of Fig. 6a-6c, but they contained slightly visible borders between the overlapping areas. [caption id="attachment_3544" align="alignright" width="177" caption="Fig. 6a Pattern grid correction. Combined from the individual measurements shown on Fig. 5a-5b. Full scale is +/- 2mm"] [13][/caption] Re-running LMA showed some improvement so I repeated  the grid re-calculation followed by LMA several times until the decrement in the calculated root mean square of the residual vector fell below the specified threshold - it took about 10 iterations to achieve this. [caption id="attachment_3546" align="alignleft" width="310" caption="Fig. 6c Reprojection errors after fitting camera extrinsic parameters and intrinsic ones assuming polynomial radial distortions, combined from the individual measurements shown on Fig. 5a-5b. Alternating X and Y differences."] [14][/caption] Simultaneously with reducing the RMS of the residual vector this procedure made all the seams caused by overlapping of individual images to fade out and the pattern grid coordinate array converged to the one shown on Fig. 6a-6c. [caption id="attachment_3543" align="alignright" width="177" caption="Fig. 6b Pattern grid correction in pseudo-colors. Full scale is +/- 2mm"] [15][/caption] Fig. 6a and Fig. 6b represent the combined pattern grid corrections similarly to the Fig. 5a and Fig. 5b for the partial grid images - the first one with the alternating horizontal and vertical corrections, the second combines both directions as a pseudo-color image. Fig. 6c shows the alternating images of the pattern grid correction as 3d plots. Coordinate correction, applied to the pattern grid reduced the root mean square of the residual vector from 0.35 to 0.14 pixels and made possible to come back to the measuring the lens distortions. Measuring and compensating the residual lens distortion As the grid correction converged to an array and at the same time the radial distortion parameters adjusted by the LMA also reached their final values, I mapped the residual differences of the pattern grid node locations acquired from the set of images to the sensor coordinates, fixed relative to the lens. [caption id="attachment_3547" align="alignleft" width="308" caption="Fig. 7 Residual lens correction after the radial distortion model, alternating for X and Y directions"] [16][/caption] That allowed me to combine such data from multiple images and measure the difference between the radial lens distortion model used with the registered pattern. Similarly as I did when combining the grid corrections, the program calculated per-image weight functions and then combined the residuals from the 15 images. When I applied the additional lens correction to each image, I had to freeze the lens distortion coefficients used by the LMA as it could become unstable because the correction array could be calculated for any combination of radial distortion parameters. The averaged correction additionally reduced the RMS of the residual vector to 0.075 pixels. Repetition of the grid correction, LMA (with fixed radial distortion parameters) and calculation of the additional lens correction was stable for the set of the images, but the result did not improve much and the final RMS for the set of images reached 0.069 pixels for the set of 15 images. This final distortion correction applied "on top" of the radial distortion correction is shown in the last column of Fig. 8, the RMS indicated (0.14px) refers to the correction amount. The same final correction is also shown on Fig. 7 as alternating X and Y corrections, in pixels. [caption id="attachment_3566" align="alignright" width="447" caption="Fig. 8 Residual lens distortion, measured by combining individual images and compensating for the pattern grid errors. The columns correspond to different degrees of polynomials of the radial distortion model, rows show the same residual with the different scale"] [17][/caption] In addition to applying correction to the lens radial distortion model after all the camera parameters were determined for the best fit, I measured the required correction after the radial polynomial model with reduced number of terms. That helped me to find out how much these terms add to the precision of the model for the particular type of lens that we are using in Eyesis cameras, what is the optimal number of polynomial coefficients in our case. The results of this comparison are shown in columns of the Fig. 8 where rows show the same functions with different full scale - from 0.375 pixels in the top to 12 pixels in the bottom row. No radial distortion (pinhole camera model) required additional correction with RMS=3.44 pix, single term (Rd/Rs term proportional to radius) reduced RMS to 1.96 pix, two terms - to 1.05 pix and the third term (commonly used, i.e. in Hugin/Panotools [18] ) reduced RMS in our case four times  - down to 0.25 pix. Adding one more term reduced the residual correction, but at this step the image almost lost the prominent so far radial structure, so just adding more terms to the radial polynomial would not help improve the approximation result. Measuring the 3D locations of the pattern grid nodes So far I used a two-dimensional model of the pattern grid even as I could see that the pattern on the wall was not flat - there is a rather thick plate that connects two halves of the pattern, same with the reinforcement near the top and wall molding near the floor. The calibration machine will rotate the camera around the center of it's optical head so most of the camera lenses (but the two specifically designed for distance measurements) will see the pattern with very little parallax and the 5-10mm non-flatness would not lead to significant errors (camera will be located 6 meters from the target pattern). [caption id="attachment_3698" align="alignright" width="158" caption="Fig.9a 3D Pattern grid correction (Full scale 2mm for X and Y, 8mm for Z corerctions)"] [19][/caption] On the other hand, as there is still a long way ahead before we'll be able to create 3d models of the nice scenery of the Southern Utah, I was tempted to do a simple 3-d reconstruction of the pattern. And we'll still need the precise 3-d model for calibration the 2 camera bottom lenses located 1 meter below the rest 24 ones. [caption id="attachment_3703" align="alignleft" width="281" caption="Fig. 9b. X and Y corrections to the pattern grid"] [20][/caption] For that purpose I added more captured images to the initial set, this time some were acquired from the tripod located some 0.8m to the left of the initial center location, and some - to the right, the total number of simultaneously processed images was 32. Next was to modify the grid correction procedure to work with 3 coordinates while each image provided only two. And I was looking to the robust method that can still work for the original set of images that were acquired approximately from the same point and it will not try to move the points far in the Z direction. The first simple modification of the 2-d target grid correction algorithm was to make each image to "vote" for the correction not in the target XY plane, but rather in the direction perpendicular to the line connecting the center of the lens with the grid point being corrected, minimal distance correction that for each camera is equivalent to the required one. Working with the set containing images from multiple points, the result correction will reduce the error in Z direction , but do it at much slower rate than calculated precisely. This method added just a few lines in the code and is still used as a fall-back, in the case of the small parallax of the images resulting in small determinant of the matrix that provided the direct calculation of the correction. That second method was minimizing the sum of squared distances from the corrected 3-d location to the lines connecting each lens location with the 2-d point on the target plane that each camera was voting for. [caption id="attachment_3702" align="alignright" width="274" caption="Fig. 9c Non-flatness of the pattern. Pattern dimensions are 3022mm by 2667mm."] [21][/caption] Both methods were followed by the normalization of the correction to reduce dependence between the effects of the correcting the grid node coordinates and moving the camera. Corrections in X and Y directions have their average values subtracted from each point, correction in Z direction is additionally compensated for tilt, equivalent to rotation of the whole pattern around vertical and horizontal axes. When the 3-d correction was applied, most of the pattern grid differences between the final grid points coordinates and the initial ideal pattern (with equally spaced nodes located on the Z=0 plane) were in the Z direction. That means that the plastic sheets on which the pattern is printed are more bended than stretched. This "redistribution" of the difference from the X and Y to Z direction is visible when comparing Fig. 6a-c and Fig. 9a-c. Large part of the residual errors in X and Y directions is caused by the mismatch of the otherwise uniform pattern halves - shift reveals itself as a step between the halves, rotation - as a different tilt of the halves on both X and Y differences graphs. It was not so easy to independently verify the measured shape of the pattern, we plan to do more measurements later, so I only measured a single profile in the center of the pattern. For that I used a laser beam parallel to the pattern surface and marked the spots on paper touching the pattern at different locations along the middle horizontal line. The precision of such quick test was about 1 mm and it did match the data from the cross section of Fig. 9c. One more experiment: camera location from the images While for the camera calibration we are mostly interested in measuring the mapping from the pixels acquired by the camera at known location to the lines lines in space, I made a separate reverse experiment to test the measurements. That was similar to a rather common task - finding the camera location in space from the available image. Contrary to usual setups where the camera distortion or even lens focal length are either completely unknown or known only approximately, and the shape of the captured objects has to be guessed, in our case the visible object (the pattern grid) was precisely measured and the camera lens was calibrated for distortions - first using the radial distortion model and then additional non-radial correction was applied. During previous measurements the model was constrained as much as possible, not only the camera intrinsic parameters and the fixed distances and angles in the kinematic model of the camera (including the tripod) were set to be common for all the images, but the location of the tripod as a whole was set not to change for the images belonging to the same group (center, right, left). Program calculated unknown parameters of the model as well as the coordinates of the pattern grid locations while these constrains were imposed. Table 1. Measured camera locations Center Right Left Number of Images 13 6 13 X position (mm) -22.55 854.00 -675.39 Y position (mm) 67.83 64.10 64.89 Z position (mm) 2322.57 2361.53 2448.79 X std. dev. (mm) 0.62 0.46 0.62 Y std. dev. (mm) 0.57 0.51 0.52 Z std. dev. (mm) 0.21 0.33 0.42 Full std. dev. (mm) 0.87 0.76 0.91 Then I reran the fitting program while "freezing" the camera intrinsic parameters and the pattern grid correction, but allowing LMA to adjust each of the 3 coordinates of the tripod position and 3 angles defining camera orientation (2 that were actually changed in the tripod head and additional camera roll) for each of the 32 images independently. That means that each (calibrated) camera location was determined by a single image. Ideally the results of these calculations should be that the tripod position would be exactly the same for all the images of the same group, but there is some discrepancy between these coordinates. The Table 1 summarizes these results, each column average and standard deviation are calculated from the corresponding subset of the images. "Full" standard deviation in the last row is calculated as a square root of the sum of squared deviations in each of the 3 orthogonal directions. [caption id="attachment_3780" align="alignright" width="369" caption="Fig. 10a Derivatives of the pixel coordinates with respect to yaw, pitch and distance (animated)"] [22][/caption] The uncertainty of the camera location along the Z axis is smaller than in X and Y directions, because camera movement along the axis causes more apparent changes in the registered image than when camera is moved perpendicular to the axis (and simultaneously rotating so the optical axis crosses the target plane at the same point. [caption id="attachment_3781" align="alignright" width="369" caption="Fig. 10b Derivatives of the pixel coordinates with respect to yaw, pitch and distance"] [23][/caption] This can be demonstrated if we consider partial derivatives of the pixel coordinates with respect to the two rotations of the camera around the fixed point on the target (that corresponds to X and Y movements) and movement to/from the camera (Z-axis). This derivatives for the center image are shown on the Fig. 10 a,b. The calculated average absolute value (rms) of the reprojection error for horizontal rotation (yaw) around the vertical axis on the target is 6.2 pix/degree, rotation around horizontal axis - 4.0 pix/degree (smaller as the camera is in landscape mode) and moving camera along the Z-axis would cause average error of 0.36pix/mm. Substituting distance to the camera (2322 mm) that results in the following average errors (per mm): X - 0.15 pix/mm, Y- 0.1 pix/mm and Z (unchanged) - 0.36 mm. Multiplying these values by the measured standard deviations in the Table 1. column for the center images result in X - 0.093 pix, Y - 0.057 pix and Z - 0.076 pix - registration precision in pixels needed to provide the measured variations in the camera coordinates. Results When we'll use the calibration machine to calibrate the cameras, both the location of the machine reference point and the coordinates of the pattern grid points will be already known - they can be measured from larger number of evenly overlapping images, and the distance from the camera to the target will be almost 3 times larger than now, so the angular errors caused by linear uncertainty will be proportionally smaller. According to the results of simultaneous fitting of the camera parameters and the target pattern geometrical properties, it will be possible to make angular measurements in the camera field of view with the precision of better than 0.2 pixels over the camera field of view - current constrained fitting of 32 images provided root mean square of combined X and Y mismatch on the sensor of 0.14 pixels (or 0.1 pixels in each direction). With the sensor 2.2 μm pixel period and 4.45 mm focal length, 1 pixel corresponds to 1.7 angular minutes, so 0.2 pixels correspond to 0.35' angle. Precision of measurements of the angles between the features in the real-life images will be limited by other factors - registration of the features that have to have enough pixels and high spatial frequency components. The precise calculation of the camera motion using the logged IMU data to compensate for the rolling shutter effect is also the project to be completed, but at least this stage - static pixel mapping of the individual camera is working and provides data to combine the mapping of the sub-cameras into the combined camera and still keep the mapping precision in sub-pixel range. We will work on minimizing the accumulation of the angular errors during "stitching" of the individual pixel maps (in each camera orientation the current pattern wills visible as 10% of the sensor area) by using the precise rotation data and possibly adding additional targets in some other directions - at shorter range but covering larger angle. All the software described in this post is written in Java as a plugin for ImageJ [24] and is available [25] on our Sourceforge project page under GNU GPLv3 license. [1] http://blog.elphel.com/2010/12/zoom-in-now-enhance-results/ [2] http://blogs.elphel.com/2010/11/zoom-in-now-enhance/ [3] http://blog.elphel.com/wp-content/uploads/2011/10/pattern-measurement.png [4] http://blog.elphel.com/wp-content/uploads/2011/10/curv-56A-rms0036-curvature-half.png [5] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_curv-56A-rms0036-curvature-half.png [6] http://blog.elphel.com/wp-content/uploads/2011/10/curv-rms00317_curvature_half.png [7] http://blog.elphel.com/wp-content/uploads/2011/10/1317925285_579597-00.tiff-flat-quater.gif [8] http://blog.elphel.com/wp-content/uploads/2011/10/curv-56A-rms0036-errors.png [9] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_curv-56A-rms0036.png [10] http://blog.elphel.com/wp-content/uploads/2011/10/nine_sm.jpeg [11] http://blog.elphel.com/wp-content/uploads/2011/10/grid-first-pass.gif [12] http://blog.elphel.com/wp-content/uploads/2011/10/grid-corr-first-pass-color.png [13] http://blog.elphel.com/wp-content/uploads/2011/10/grid-diff.gif [14] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_a20-grid-difference-1pix-1mm.gif [15] http://blog.elphel.com/wp-content/uploads/2011/10/grid-diff-color.png [16] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_Cumulative_chn_0_corrections.gif [17] http://blog.elphel.com/wp-content/uploads/2011/10/distortion-orders.png [18] http://hugin.sourceforge.net/docs/manual/Lens_correction_model.html [19] http://blog.elphel.com/wp-content/uploads/2011/10/grid44-minus-default.gif [20] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_grid44-minus-default.gif [21] http://blog.elphel.com/wp-content/uploads/2011/10/Surface_Plot_of_grid44-minus-default-Z.png [22] http://blog.elphel.com/wp-content/uploads/2011/10/derivatives-5.gif [23] http://blog.elphel.com/wp-content/uploads/2011/10/derivatives-5.png [24] http://rsbweb.nih.gov/ij/ [25] http://elphel.git.sourceforge.net/git/gitweb.cgi?p=elphel/ImageJ-Elphel

We made a new Live USB version of Elphel Toolkit. [1] It is available for downloading here [2]. Software It is an entire Operating System that can be booted from a USB drive or DVD (of course you can install it on your computer as well) and comes with all Elphel relevant software preinstalled. As the basis we chose Kubuntu 10.04 LTS [3] (Long Term Support distribution) 32-bit which is supported by its developers until April 2013. Everything described on our Software Kit wiki page [4] is preinstalled. That allows you to update the camera firmware (reflash), play a live video-stream from a camera and also build a custom firmware image for the camera. Some of the installed software: VLC/MPlayer CVS/GIT UNFS3 (network sharing - used in update camera's firmware process) Axis ETRAX Cross Compiler (writing and compiling C-based applications for the camera) KDevelop 3.5 Icarus Verilog & GTKWave (FPGA verification) Note: Xilinx ISE WebPack [5] is not installed. Should be downloaded and installed personally. It is only needed for creating FPGA bitstreams. Burning the ISO The iso-image can either be burnt on CD/DVD or written on a boot-able USB thumb drive. Here are couple programs that can be used to create a live USB thumb drive: Startup Disk Creator (Linux only) - a default (K)Ubuntu application.* UNetbootin [6] (Linux/MacOSX/Windows) - also free. *- The image is written fine but be aware that currently there is this bug [7] that has a fix [8]. Or you can just work around the bug by typing "help" and then hitting "enter" after a menu appears. Note: Sometimes you need to enable booting from USB in BIOS. Links Elphel Live Toolkit Kubuntu 10.04 LTS based USB/DVD ISO file [9] Elphel Software Kit wiki page [10] Elphel wiki live USB page [11] Firmware Images [12] Camera firmware update manual [13] [1] http://blog.elphel.com/2011/10/live-usbcddvd/liveusb_screenshot/ [2] http://community.elphel.com/files/live-dvd/ [3] http://www.kubuntu.org/ [4] http://wiki.elphel.com/index.php?title=Elphel_Software_Kit_for_Ubuntu [5] http://www.xilinx.com/products/design-tools/ise-design-suite/ise-webpack.htm [6] http://unetbootin.sourceforge.net/ [7] https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/645818 [8] http://xiaonosekai.wordpress.com/2011/09/02/ubuntu-usb-boot-problem/ [9] http://community.elphel.com/files/live-dvd/ [10] http://wiki.elphel.com/index.php?title=Elphel_Software_Kit_for_Ubuntu [11] http://wiki.elphel.com/index.php?title=Live_USB [12] https://sourceforge.net/projects/elphel/files/elphel353-8/ [13] http://community.elphel.com/files/live-dvd/readme.html

We have finally received the parts for Elphel-Eyesis-4π camera and started assembling them hoping that all will fit together as we planned. And for the most part they do, which seems a bit like magic to us: you design the camera on the computer in a 3D CAD program, make a long list of parts it will consist of and then a couple months later it all turns into physical object, not just a virtual 3D design. Of course some of the parts will need minor modifications - some are due to mistakes made by us, and some are manufacturing problems. But none of them were significant enough to prevent us from assembling the first 3 prototypes, that will be 100% operational spherical panorama cameras. Elphel-Eyesis-4π is the second generation Panoramic Imaging System by Elphel Inc. It is able to capture high-resolution images in full 360 degrees and create 4π (in steridians) spherical panoramas at a high frame rate. The actual recording device consists of a weatherproof camera head that contains the image sensor front-ends and lenses in spherical distribution to cover the entire 360 degree area. The rest of the electronic components as well as the SSDs for data storage are contained inside the camera pole. Elphel-Eyesis-4π covers a full sphere. 24 sensors (8 in horizontal array, 8 pointing at +30 to + 90 degree (zenith) and 8 pointing at -30 to -90 degree (nadir) ensure a uniform high resolution distribution over the entire covered area. A new Internal Measurement Unit (IMU) mounted at the top of the camera pole provides high resolution 3D position and orientation of the camera. [caption id="attachment_3023" align="alignnone" width="177" caption="Fig.1. Test assembly."] [1][/caption] [caption id="attachment_1791" align="alignnone" width="177" caption="Fig.2. 3D CAD rendering."] [2][/caption] The main parts of the camera are the head and the pole. Inside the head there's the optical part consisting of the external Head Enclosure with interchangeable UV protection filters to prevent physical damage of the internal parts and keep the lenses clean. The internal part of the head holds the sensor-lens modules. The GPS receiver and the Inertial Measurement Unit (IMU) are mounted on the top of the camera head. The IMU's platform sits on rubber spacers (blue colored on Fig. 4) acting like dampers to reduce noise in acceleration and angular speed measurements. The GPS is placed above the IMU and is protected with an additional plastic cover. The camera shade (Fig.5) which wraps around the camera head consists of 2 halves which are assembled from a number of horizontal and vertical fins that are welded together and then anodized. Both the camera shade and camera head are designed as 2 halves for more convenient assembly. The camera pole is designed to be both lightweight and solid support structure at the same time. Inside the weather sealed camera pole there are camera system boards and Solid State Drives (SSDs). Fig.1 and Fig.2 show the inside of the pole. The electronics are placed inside the pole like floors are laid out inside a skyscraper building. There are 4 floors with 2 system boards each (8 total) and 2 more floors with 4 SSDs each  (8 total, depending on the configuration). Additional support structures at each "floor" give the pole additional structural stability. Camera dimensions are: Head with Shade: 230mm Diameter by 308mm Height (including IMU and GPS). Camera Pole: 780mm - 1025mm (the longer pole is to include 2 additional camera modules at the bottom of camera pole, for measuring distance to photographed objects). [caption id="attachment_2980" align="alignleft" width="324" caption="Fig.3. parts with a 300mm ruler and CD for scale."] [3][/caption] [caption id="attachment_3007" align="alignleft" width="370" caption="Fig.4. IMU platform and GPS plasic cover."] [4][/caption] [caption id="attachment_3288" align="alignleft" width="370" caption="Fig.5. camera shade before welding and anodizing."] [5][/caption] [1] http://blog.elphel.com/2011/10/elphel-eyesis-4%cf%80-preassembly-stage/eyesis4pi_2-2/ [2] http://blog.elphel.com/2011/02/eyesis-4pi/eyesis4pi-internals/ [3] http://blog.elphel.com/2011/10/elphel-eyesis-4%cf%80-preassembly-stage/some_parts_4/ [4] http://blog.elphel.com/2011/10/elphel-eyesis-4%cf%80-preassembly-stage/top_with_imu_and_tubes/ [5] http://blog.elphel.com/2011/10/elphel-eyesis-4%cf%80-preassembly-stage/shade/

[1] [2] We are proud to add a new product and camera KIT to the Elphel portfolio. See the pricelist [3]. The NC353L-369-IMU/GPS is a new camera configuration with an Inertial Measurement Unit (IMU) and optional GPS receiver. In addition to storing the geographical coordinates with each captured image in a video stream this allows to also save 3D orientation (yaw, pitch and roll) and 3D acceleration (Six Degrees of Freedom Inertial Sensor) of the camera at the moment of capturing an image at very high precision (2400 samples/second). A detailed description can be found in the previous post [4]. [1] http://blog.elphel.com/2011/09/nc353l-369-imugps/nc353l-369-imu-gps_4/ [2] http://blog.elphel.com/2011/09/nc353l-369-imugps/nc353l-369-imu-gps_1/ [3] http://www3.elphel.com/price_list#KIT-NC353L-369-IMU.2FGPS [4] http://blog.elphel.com../2011/05/imu-and-gps-integration-with-elphel-cameras/

[caption id="attachment_2740" align="alignleft" width="500" caption="WebGL Panorama Editor (view mode)"] [1][/caption] This April we attached Eyesis camera to a backpack [2]and took it to the Southern Utah. Unfortunately I did not finish the IMU hardware [3] then so we could rely only on a GPS for tagging the imagery. GPS alone can work when the camera is on the car, but with a camera moving at pedestrian speed (images were taken 1-1.5 meters apart) it was insufficient even in the open areas with a clear view of the sky. Additionally, camera orientation was changing much more than when it is attached to a car that moves (normally) in the direction the wheels can roll. Before moving forward with the IMU data processing we decided to try to orient/place some of the imagery we took manually - just by looking through the panoramas, adjusting the camera heading/tilt/roll to make them look level and oriented so the next camera location matches the map. Just tweaking the KML file with the text editor seemed impractical to adjust hundreds of panoramic images so we decided to add more functionality to our simple WebGL panorama viewer [4], make it suitable for both walking through the panorama sets and for the editing orientations and locations of the camera spots. WebGL and large textures Our first panorama textures were both small (8192x4096 pixels) and dimensions were power of two - initially I could not make it work with Non-power of two (NPOT) textures. The actual Eyesis panoramas are larger (resolution is set to match camera pixels at the equator) , so that initial texture size was good just for the first experiments. The problem with the NPOT was rather easy to solve, solution is described on the Khronos.org wiki [5], not just the minification filter should not use mipmapping (I made sure it did not before), but repeat mode should set to CLAMP_2_EDGE (I still do not fully understand that as we do not use any repeated textures).  Anyway, couple extra lines of magic did the trick: gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE); So it did work on my computer with 14,272x7,136 images. Worked on the computer I was using at the moment, but not on the two others we tried - still black screen instead of the nice scenery. So probably those other computers had less capable graphic card, so we tried to investigate - where is the limit. Is it the size of individual textures or the total amount of the texture data transferred to the graphic card memory. The viewer had the capability to specify number of simultaneously loaded panoramas, but changing that parameter did not seem to make any difference. So next thing was to split the single image that was put on the sphere, as in the nice WebGL lesson [6] into several tiles. To make it easier to experiment I modified the sphere model to use variable number of tiles and just splitting the full 14,272x7,136 into two square ones of 7,136x7,136 made it work on those two computers that failed initially - the limit seemed to be at 8192 as the tile dimension in each direction,  anything 8192x8192 or under seemed to work on our other computers of different types. To play it safe we split images once more in each direction to the total of 8 tiles, adding suffixes to the filenames, i.e. texture.jpeg was split into texture_1_2.jpeg and into texture_2_2.jpeg (1 of  2 and 2 of 2), or texture_1_8.jpeg, ... texture_8_8.jpeg ( 1 of 8, ... 8 of 8 ) - with such naming we could keep different tiling files in the same directory and have just the original (texture.jpeg) specified in the KML file. For the first experiments I just used GIMP to split the images, but later Oleg made a script with ImageMagick to do it automatically. One pixel size is small compared to the full image, but correct tiling should repeat the border pixels between the next tiles, same is true for the first and last pixel rows/columns - they should "roll over". And as JPEG images are not always correctly processed when the image size is not multiple of 8 (block size), the tile size before adding extra row/column should be 8*n-1, so we adjusted the Hugin [7] output accordingly and used TIFF (no problems with arbitrary image size) as the output. The tiling script was taking the full panorama tiff (14268x7134), splitting it into 3567x3567 images and adding extra row and column (from the next image) to the output of 3568x3568 tiles. If the tiling does not repeat the border pixels it will be clearly visible on the panorama interpolation when the zoom is set high enough. [caption id="attachment_2745" align="alignright" width="333" caption="Oleg with Eyesis in a canyon"] [8][/caption] An old method of orientation -- use the Sun While we could use (to some degree of success) the GPS data for orienting the camera when mounted on a car (camera is rigidly attached to the car, and car is pointed from the current camera location to the next one when going straight), it is much more difficult with a backpack -  the "forward" direction of the camera is just somewhat related to the vector connecting camera positions, we also noticed that in many cases the camera was significantly tilted sideways, not just forward that was inherent to the simple backpack mount we made. Luckily for us it is usually sunny in the Southern Utah and we know the location and the time when the images where captured, so it is possible to calculate where (azimuth, elevation) the Sun should be visible. Quick search got us to the NOAA Sun Position Calculator [9] that has all the calculations in the Javascript code. So Sebastian just extracted the web page code to a sunpos.js file that we could incorporate into the software for panorama set editor. Initial testing failed - the Sun azimuth was to the North and elevation - negative, then I looked at the original NOAA page and found that it was expecting longitude to be positive for the West and negative - for the East. Of course, it is an American agency, but I still expected the sign of the longitude to be opposite to that. That problem fixed and we had a nice spot on many of the images that we could use when orienting the panoramas. Processing the Imagery: Little Wild Horse Canyon As soon as the software got some functionality we started testing it on the two sets of the images. One was made in a very narrow canyon where GPS gave up at the very entrance, no view of horizon, even the Sun was obscured in most places but the task was somewhat simplified by the fact [caption id="attachment_2741" align="alignleft" width="500" caption="WebGL Panorama Editor (edit mode)"] [10][/caption] that the camera path was essentially a "2-d" one - the small processed segment has virtually constant (slowly increasing) altitude.  Images were captured at just 1.25 frames per second, but the compression quality was set too high  - in some places the internal image memory buffer was overrun and a few panoramas were lost, it happened in the narrow places where all the sub-cameras field of view were filled with detailed rocks, compressed file sizes went above 2.5MB for 5MPix sub-images. This set consists of 164 individual panoramas captured on a small (~150m) segment of the canyon. The images were captured with odd/even alternating exposure settings (4:1 ratio). As the camera was moving it is not possible to combine those images into HDR ones, they only can be used to switch manually to the close by spot shot at different conditions. Eventually we plan to use such alternating brightness images in the future 3-d reconstruction - in that case it will be relatively easy to combine both image sub-sets into a single HDR 3-d model. The images show some of the problems with the first generation Eyesis camera designed to be mounted on a car when used attached to a backpack. One quarter of the whole sphere is lost around nadir (below 30 degrees from the horizon), the top quarter is covered by a single fisheye lens having much smaller resolution than the eight side-pointed sub-cameras. Those problems were exaggerated by the severe tilting of the camera (in some places by nearly 30 degrees) that raised the nadir spot to near horizon, same with the low-resolution top camera.  When processing the images we paid less attention to the fisheye camera - in the next Eyesis-4π [11] camera all the channels will have the same high resolution, they will all have the same angular resolution. The "nadir hole " will also be covered by additional 8 sensors pointed downwards. Processing the Imagery:  Goblin Valley [caption id="attachment_2742" align="alignright" width="500" caption="WebGL Panorama Editor (aligning the two panoramas shown)"] [12][/caption] The other image set was captured in the Goblin Valley - GPS showed meaningful data most of the time (lost just once when too close to the rock wall), just the altitude was unusable. It was rather easy to level the panoramas using the Sun and the far features, those features could be used for aligning panoramas to each other - selecting one as a reference and then mixing two panoramic images by adjusting their transparencies with a slider. And when matching the far features on the pairs of the panoramas, there were small (~1-2°) errors visible. Some of those errors were caused by the imperfection of the Hugin templates we created (in the final software we'll have precise pixel mapping, so we considered the manual stitching as a temporary solution), others were due to the rolling shutter (ERS) effect in the heavily rotating camera. We'll be able to separate those effects and mitigate them when we'll finish the static optical distortion calibration of the camera lenses and use the IMU data to measure camera rotation during image acquisition. There also image processing artifacts visible. We did not use any regular de-noise filters included with most image processing/editing programs to evaluate our aberration correction program itself. Most visible artifacts  are caused by the top fisheye ("noisy" sky). The main source is the underexposure of the fisheye channel. Fisheye lens has higher F-number than the side ones and the current software applied the same exposure value to the group of the 3 sensors, so the pixel value range for that channel was only a half of that of the other channels. The sharpening process amplifies the noise - and the filtering parameters were adjusted for the fisheye lens not as good as the ones for the other lenses (there are some known improvements left to be done for them also). The poor quality in the top part of the panoramas was one of the reasons we decided to use different optical layout for the Eyesis-4π [13], so this is just a temporary problem. Supplementing IMU with the two additional sub-cameras [caption id="attachment_2743" align="alignleft" width="277" caption="A pair of additional camera modules to supplement the IMU data"] [14][/caption] While working on the IMU integration [15] with the Eyesis camera, we were thinking of supplementing the inertial data with the additional optical sensors (just extra camera sub-modules identical to the ones used for the panorama image acquisition). The 3-d reconstruction we plan to implement will combine images made from the different camera locations while it is moving, not from the stereoscopic  system.  For that we need precise knowledge of the camera relative position and orientation,  and for the CMOS sensors prone to the ERS distortion we need to know the camera position/orientation during each frame acquisition. Most of the position measurement will be done using the IMU data logged at 2400 samples/sec, but we'll have additional pair of camera subunits located inside the camera tube body (73mm diameter) that is used to carry camera electronic modules and SDD data storage, near the lower end of it, approximately one meter below the main camera sensors. The pair of additional lenses will be pointed sideways and have FOV overlap with the cameras above them. With the sensors oriented in "portrait" mode (vertical scan lines) and synchronized scanning the same features on the main and additional sensors will be captured at the same time - that will simplify distance measurement when the whole camera is in the motion and rotation. Correlating the same features in two (or more) stereo pairs (consisting of the main and additional images) will provide the relative positional information to be combined with the inertial data and used for the 3-d reconstruction in the other areas, not just in the overlap FOV.  Two opposite-pointed camera modules (instead of just one) improve the measurement precision (by reducing partial ambiguity between movement and rotation) and help in the cases when the view in one direction is obstructed (i.e. by the moving traffic). Of course, sometimes both views can be obstructed by the moving traffic, but we will still have the inertial and GPS measurements available. So for the planned 3-d reconstruction we do not need to double the number of sensors and have stereo-panoramic camera - it is sufficient just to have stereo measurements in some areas for precise positional data. Links to the panorama sets The program (webgl_panorama_editor.html) uses (and modifies in edit mode) the KML file that keeps each camera location, orientation, comments, acquisition time stamp and the image link (actual image filenames are modified during tiling). Additionally this file holds the "3d visibility ranges" - this data, the view only/edit mode is determined by the file permissions, if it is read only - program works as a viewer, if write is permitted - it works in the edit mode.  This file is specified as a parameter in the URL string, i.e. ...&kml=map_goblins_06.kml ...&kml=map_goblins_06_readonly.kml (copy of the map_goblins_06.kml with writing disabled) It is possible to specify additional kml file that is used to overwrite some settings of the main one, i.e. &mask=map_goblins_mask_light.kml (or &mask=map_goblins_mask_dark.kml or  &mask=map_goblins_mask_all.kml) The program uses image names as keys and overwrites all the original KML file values in the response from the back end PHP script (map_pano_02.php). In the viewer mode it is used just to specify visibility of the alternating exposure nodes - make all visible (*_light.kml), only those acquired with long exposure time (*_light.kml) or those with the short - *_light.kml . There are more settings that can be passed via the URL parameters,  they can be generated from inside the program itself. So here are the viewer (read only) mode links: 1. Goblin Valley set (83 panoramas) - dark, light and all frames : webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_dark.kml&azimuth=230 [16] webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_light.kml&azimuth=230 [17] webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_all.kml&azimuth=230 [18] 2. Little White Horse Canyon set (164 panoramas) - dark, light and all frames : webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_dark.kml&azimuth=90 [19] webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_light.kml&azimuth=90 [20] webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_all.kml&azimuth=90 [21] Editor mode: Editor mode involves creating a copy of the original kml file that you can modify. The directory http://community.elphel.com/files/eyesis/webgl-panorama-editor/kml_files/ [22] is open for writing, you can start with the same KML file as in the viewer example or with one of the files in the kml_files directory (the mask file if present should match the data set - Goblins Valley or Little Wild Horse Canyon). You may specify a name for the new kml file (the name should start with "kml_files/" and end with ".kml") and use "&proto=map_goblins_06_readonly.kml" (or "&proto=map_lwhc_14_readonly.kml") for the original KML file that will be copied to the new one, i.e. webgl_panorama_editor.html?proto=map_lwhc_14_readonly.kml&map=all&kml=kml_files/my_new_lwhc.kml [23] That link will copy "map_lwhc_14_readonly.kml" to a new file that you can modify - "kml_files/my_new_lwhc.kml". If "kml_files/my_new_lwhc.kml" already exists, it will just use it as is without copying "map_lwhc_14_readonly.kml" over, if the file copying would fail, the original file (in this example - "map_lwhc_14_readonly.kml") will be used in read only mode (editing controls disabled), that can happen if the specified new KML file name does not match the requirements (i.e. is not inside "kml_files" directory, does not end with ".kml" or just is not a good name for the operating system). You may also verify that the new file is created (or download it) by opening http://community.elphel.com/files/eyesis/webgl-panorama-editor/kml_files/ [22] page in the browser. It is also possible to specify the file names inside the program by opening the "Settings" dialog - just enter the "kml_files/*.kml" in the KML file input field and the original file into  the KML prototype one. Similarly, to edit Goblin Valley panorama set, you may use the following link (just modify the name of a new KML file): webgl_panorama_editor.html?proto=map_goblins_06_readonly.kml&map=all&kml=kml_files/my_new_goblins.kml [25] The code is in Elphel Sourceforge Git repository - http://elphel.git.sourceforge.net/git/gitweb.cgi?p=elphel/webgl_panorama_editor;a=tree [26] [1] http://blog.elphel.com/wp-content/uploads/2011/06/htr_91.jpeg [2] http://blog.elphel.com/2011/04/panoramas-from-the-backpack/ [3] http://blog.elphel.com/2011/05/imu-and-gps-integration-with-elphel-cameras/ [4] http://blog.elphel.com/2011/01/experimenting-with-webgl-panoramas/ [5] http://www.khronos.org/webgl/wiki/WebGL_and_OpenGL_Differences [6] http://learningwebgl.com/blog/?p=1253 [7] http://hugin.sourceforge.net/ [8] http://blog.elphel.com/wp-content/uploads/2011/06/oleg_eyesis_5680.jpeg [9] http://www.srrb.noaa.gov/highlights/sunrise/azel.html [10] http://blog.elphel.com/wp-content/uploads/2011/06/htr_21.jpeg [11] http://blog.elphel.com/2011/02/eyesis-4pi/ [12] http://blog.elphel.com/wp-content/uploads/2011/06/htr_81.jpeg [13] http://blog.elphel.com../2011/02/eyesis-4pi/ [14] http://blog.elphel.com/wp-content/uploads/2011/06/e3d_041.jpeg [15] http://blog.elphel.com/2011/05/imu-and-gps-integration-with-elphel-cameras/ [16] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_dark.kml&azimuth=230 [17] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_light.kml&azimuth=230 [18] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_goblins_06_readonly.kml&mask=map_goblins_mask_all.kml&azimuth=230 [19] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_dark.kml&azimuth=90 [20] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_light.kml&azimuth=90 [21] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?kml=map_lwhc_14_readonly.kml&mask=map_lwhc_mask_all.kml&azimuth=90 [22] http://community.elphel.com/files/eyesis/webgl-panorama-editor/kml_files/ [23] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?proto=map_lwhc_14_readonly.kml&map=all&kml=kml_files/my_new_lwhc.kml [24] http://community.elphel.com/files/eyesis/webgl-panorama-editor/kml_files/ [25] http://community.elphel.com/files/eyesis/webgl-panorama-editor/webgl_panorama_editor.html?proto=map_goblins_06_readonly.kml&map=all&kml=kml_files/my_new_goblins.kml [26] http://elphel.git.sourceforge.net/git/gitweb.cgi?p=elphel/webgl_panorama_editor;a=tree

[caption id="attachment_2600" align="alignleft" width="300" caption="CAD rendering of the Analog Devices ADIS-16375 Inertial Sensor, 103695 interface board and 103696 serial GPS adapter board attached to the top cover of NC353L camera"][/caption] For almost 3 years we had a possibility to geo-tag the images and video using external GPS and optional accelerometer/compass module that can be mounted inside the camera. Information from the both sensors is included in the Exif headers of the images and video frames. The raw magnetometer and accelerometer data stored at the image frame rate has limited value,  it needs to be sampled and processed at high rate to be useful for the orientation tracking , and for tracking position it has to be combined with the GPS measurements. We had developed the software to receive positional data from either  Garmin GPS18x (that can be attached directly to the USB port of the camera) or a standard NMEA 0183 compatible device using USB-to-serial adapter. In the last case it may need a separate power supply or a special (or modified) USB adapter that can provide power to the GPS unit from the USB bus. When working on the Eyesis panoramic camera [1] we realized that we need higher precision position and orientation, for the GPS we need more than one sample per second, and we also looked for the synchronization of the measurements to the timing signal available on many units, some of our users were also asking for that option. We started looking for the turn-key solution - device that has the real-time processing capabilities of the inertial data, combines it with that from the attached GPS unit, in some cases has manometer for the atmospheric pressure measurement (and so the altitude) and an input for the odometer pulses from the wheel sensor on the vehicle. Such devices significantly simplify the processing on our side, but internally they run closed, proprietary software so we might have troubles in the future trying to adjust the control parameters to the particular operation conditions or to combine the data being used by such units with the additional measurements specific to our case. At the same time I realized that we do not need the position/orientation data in the real time, we can just record it while acquiring images and then later process the measurements when processing the images themselves - I wrote about it in the earlier post [2]. This approach gives more flexibility for the sensor selection, we decided to start with the high resolution 6-DOF sensor manufactured by Analog Devices - it has 16-bit resolution of the internal ADC, 330Hz of the sensor bandwidth and 2.4kHz sample rate. The IMU adapter board is designed to support different sensors, ones with different power supply voltage and different SPI clock rate  i.e. ADIS-16405 that has lower resolution than ADIS-16375, but includes magnetometers. Sampling multiple registers at 2.4kHz requires recording of some 100 kilobytes per second, the data has to be buffered by hardware so the camera CPU would not have to service low-latency interrupts - the CPU already has to provide various functionality related to the image capturing and recording, control of the acquisition modes, it has to communicate with the host computer and/or with the other cameras. Those requirements were suggesting to use some of the remaining camera FPGA resources - unlike software implementation such approach causes no interference with the existent functionality ( until all the FPGA logic cells, registers and memory are exhausted, of course). Logged data is buffered in the FPGA on-chip FIFO memory and then transferred to the system memory using the DMA method . When the 353 camera was designed I connected two external DMA channel signals from the FPGA to the CPU, but so far only one channel was used, so that was an opportunity to put the second dormant one to do some useful work. Cameras use 64-bit timestamps internally (32-bit seconds from the epoch and microseconds), these timestamps are attached to each logged sample, the sample size is fixed to 64 bytes including 8 for the timestamp and 56 for the payload, that allows to record up to 28 of the IMU internal 16-bit registers or 14 of the 32-bit ones (the set of the sampled registers is run-time programmable). When the IMU logger Verilog code [3] was written, simulated and tested in the camera, it was very tempting for me to add logging of additional data, not just the IMU. And when making the GPS interface board (basically USB-to-Serial adapter with GPS power supply provisions and sync pulse input connected to the FPGA GPIO) I already connected the RxD line to another FPGA GPIO line - two such lines are available on each extension port of the 10369 board. That gave the FPGA capability to listen to the GPS data while the full duplex communication between the GPS and the CPU is preserved. Unfortunately some of the NMEA 0183 sentences may be longer than the 56-byte payload of the logger (the GPS data rate is much lower than that of the IMU so I tried to reuse the logger format optimized for the IMU), so I added some minor processing of the NMEA sentences during logging (for what I was criticized on #elphel IRC channel [4] ). So current code can recognize (according to run-time specified pattern) up to four different $GPxx sentences and extract data according to per-sentence provided format: each field can be either a string of the 7-bit characters (last byte is stored OR-ed with 0x80) - such fields are usually a single-character ones like "E" for East or a number, that may include decimal point and "-" sign. Such data is encoded four bits per character (so two characters per byte) with 0xf added for the field termination. Such compressed format allows to log any sentence, the full sample includes 8 bytes of the timestamp (the normally unused high microsecond byte is now used to store the logged channel number), followed by the half-byte of the sentence number (one of four) followed by the actual sentence data - up to 55.5 bytes that corresponds to about 100 characters of the raw NMEA. Additional configuration parameters select the source of the synchronization - the moment when the timestamp is sampled. It can be the GPS sync output (if available), the start of the first character in the serial data after the pause or the start of each individual NMEA sentence. Third source of the logged data is the inter-camera synchronization module. In the multi-camera systems (like Eyesis) there is a hardware synchronization between individual camera modules, the "master" camera sends a trigger signal simultaneously to each of the "slave" ones, for simplicity of the overall synchronization process (for each camera to determine which frame number this particular synchronization pulse match to) the synchronization signal encodes the full 64-bit timestamp of the "master" camera, this timestamp is normally recorded with the image instead of the local timestamp. So the logger is capable of recording a pair of the timestamps for each image acquired - the "master" timestamp (stored in each camera image) and the local one. That provides means to match IMU/GPS data to the "master" clock time. Last (at least for now) source of the logged data is "odometer" channel (or it can be something else, of course). The external sync port of the NC353 camera has an output that can be used as a power source for an LED and an input that can be connected to a photodiode. These signals can be used for the reflective sensor of the wheel rotation, providing the system with odometer data. It is possible to write the additional message text (up to 56 bytes) by the software (i.e. received over the network) to the FPGA internal memory - this data will be logged at the next pulse. Odometer pulses can use the software input - in most cases couple millisecond jitter that exists for the network synchronization is not critical in this case, so the channel can log pure network messages. The software driver [5] configures the default logger parameters and the associated DMA channel, provides the means to modify the settings and reads the DMA memory buffer when the data is available, so a just cat /dev/imu >output_file is all what is needed to record the data from the IMU, GPS (with optional time sync output), odometer and inter-camera synchronization module to the storage file. This functionality will be a part of the Eyesis-4pi system, we also plan to use it with the NC353 cameras. The initial revision of the assembled boards is tested and we placed an order for the final ones with some minor bug fixes, we ordered the modified top cover for the camera with extra recesses and holes to accommodate this new hardware (see the CAD rendered image above). We plan to post the raw logged data with the Eysis images in the next couple weeks. [1] http://www3.elphel.com/eyesis [2] http://blog.elphel.com/2011/02/eyesis-4pi/ [3] http://elphel.cvs.sourceforge.net/viewvc/elphel/elphel353-8.0/fpga/x3x3/imu_logger.v?view=markup [4] http://irc.elphel.com/irclogs/elphel.2011-05-06.log.html#t13:05:33 [5] http://elphel.cvs.sourceforge.net/viewvc/elphel/elphel353-8.0/os/linux-2.6-tag--devboard-R2_10-4/arch/cris/arch-v32/drivers/elphel/imu_log353.c?view=markup

[caption id="attachment_2205" align="alignleft" width="300" caption="Eyesis Backpack with Netbook for initial set-up"] [1][/caption] Tacking high-resolution panoramic images in the remote places, that can only be accessed by feet is an option now available with the Elphel-Eyesis [2] 360 degree panorama camera. The camera's size (1.3 x 0.3 meters) and relatively light weight (10kgs) allow to mount it on a backpack frame and carry by a person. There were multiple requests for the backpack option by our customers since the development of Elphel-Eyesis camera, but other projects were of higher priority, until this spring, when we finally decided it was time to take Eyesis hiking.  After all, we have worked hard on this project for many months, so we ought to have some fun with it too, and take panorama images of the places we knew and enjoyed for it's scenery. On April 19th, 2011, we took the Elphel mobile office to camp for 5 days in Southern Utah near the Goblin Valley State Park to try out Eyesis in beautiful places not yet available in continuous panoramic imagery, mainly because of their inaccessibility for car, ATV, or even a tricycle. Hiking with Eyesis in the backpack (30 lbs/14 kg total weight with the battery pack) allows us to capture a continuous stream of geotagged (GPS) 360° panoramic images. With the current battery pack (just a regular UPS with lead battery) we can take up to one hour and forty minutes of footage at a rate of 5 frames per second. [caption id="attachment_2204" align="alignnone" width="480" caption="Route #1 - Along the Creek"] [3][/caption] Our first test hike was around our camp, along the creek, and into a tiny, 20 meters long, canyon. The green dots mark the route and points 1 and 2 show the location of 2 panoramas presented. [caption id="attachment_2206" align="alignleft" width="170" caption="hands-free operation even allows rock climbing with Eyesis"] [/caption] After the initial set-up was completed on a small and lightweight EeePC we started the recording and disconnected the computer again, as it is better to have hands free while hiking, or even rock climbing. All images and corresponding metadata are stored on Eyesis' internal SSDs automatically. Photograph on the right had been tilted and clipped to make it look like we are actually rock climbing with Eyesis. It is a joke, of course. [caption id="" align="aligncenter" width="674" caption="Image 1: Exiting the tiny canyon (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] It's quite difficult to keep an upright position all the time (especially in a steep terrain) to ensure properly leveled panoramas. Climbing up or down-hill will result in a image like this: [caption id="attachment_2208" align="alignleft" width="674" caption="Image 2: Elphel Campsite (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] Current development of the IMU (Inertial Measurement Unit) logger, among other features described in Eyesis 4Pi [4] article, will also help to overcome the tilt problems. Our original plan was to have the IMU support ready for this trip, - may be you can see Andrey inside the mobile office working on it, but the time was limited and we end up not using IMU on this trip. All panoramas you see here are oriented manually. Hiking Among the Goblins (Or is it Mars?) The next day we ventured away from the camp, taking Eyesis to the Goblin Valley State Park. The GPS was working fine, again and recorded the trail where we made panoramas. Once all the panoramas will be processed we will have them available as a virtual tour via the Elphel Panorama Viewer. [caption id="attachment_2224" align="alignleft" width="480" caption="Route # 2 - Goblin Valley"] [5][/caption] Points 1, 2 and 3 show the locations of panoramas we present here. [caption id="attachment_2219" align="alignleft" width="300" caption="Goblins and ..."] [6][/caption] [caption id="attachment_2220" align="alignnone" width="300" caption="a Martian"] [7][/caption] [caption id="attachment_2229" align="alignleft" width="674" caption="Image 1: Bright sunny day in Goblin Valley (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] [caption id="attachment_2230" align="alignleft" width="674" caption="Image 2: Dramatic Shadows (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] [caption id="attachment_2231" align="alignleft" width="674" caption="Image 3: Stone formations gave valley it's name (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] The Little Wild Horse Canyon The Little Wild Horse Canyon is very narrow. At times, the canyon is the width of a person. Since the Eyesis camera can focus on objects no closer than four meters, we show the panoramas that were made outside the narrow slot canyon. [caption id="attachment_2237" align="alignleft" width="480" caption="Route #3 - Little Wild Horse Canyon"] [8][/caption] The satellites were out of sight most of the time, so the GPS data was not as accurate  in this narrow winding canyon. Again, the IMU (and other distance measurement sensors, that we plan to add in Elphel 4Pi camera) will mitigate these problems. However, the panoramas turned out spectacular. [caption id="attachment_2238" align="alignleft" width="480" caption="What is that? - A rocket launcher!"] [9][/caption] It is a very popular hike, so Eyesis got a lot of attention from visitors - almost as much as the canyon itself! [caption id="attachment_2239" align="alignleft" width="674" caption="Image 1: Approaching the narrow slot (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [/caption] [caption id="attachment_2241" align="alignleft" width="674" caption="Image 2: The Canyon kept winding (Panorama Viewer requires a browser supporting WebGL like Firefox4 or Chrome)"] [10][/caption] Update (by Andrey) There is a nice discussion of these panoramas on a PanoToolsNG forum [11]: "That's of course just my snobbish sight on it because I can't see a reason why one would want to take the pain to carry such a thing just to bring horrible pictures from beautiful places to the rest of the world." What I saw was not too impressive; lack of focus, poor stitching, uneven horizon, huge nadir hole, all for over $24,000! or did I miss something? Well, these panoramas are not perfect, of course. But I got an impression that it is an apples-to-oranges comparison (you may find more an apples-to-apples one in our previous blog post [12]). The panoramas we tried here were made with a rig designed to be mounted on a car roof, the backpack version was a quick test (we actually planned this trip to try camera on ATV, but we had to change our plans for the reasons out of our control), so nothing was done to make a camera vertical - that would need either angled mount of the optical head or (as a quick fix) a counter-balance attached to the camera (something like a carrot on an stick extending in front of an operator). That tilt caused the black nadir hole being asymmetrical, you may want to look at our project under development - Eyesis-4pi [13] - it is designed to have the same resolution in the zenith and a zero nadir void -- but there still will be some vehicle or person visible, unless the camera will be in a free fall (i.e. in space) as we were already suggested :-) . The camera we used simultaneously acquires all 9 images in a way that there are no voids even when the camera is moving fast, so we acquired thousands of panoramas during each of the trips, the acquisition rate was set to about 2 frames per second. Yes, you definitely can make artistic panoramas using a single inexpensive camera using adapter that keeps the entrance pupil of the lens in the same point, but that would not work for the fast moving camera - so our only tool to keep the parallax low was to design the camera to have components as close as possible - but the parallax would never be zero Poor focus? Well, this may be true for the very close objects, the camera is designed for the relatively high distance to the objects. The focus is fixed, of course, and the lenses are rather high power to avoid motion blur when the camera is moving at highway speed. So with f=4.5mm and F=2.0 the near limit of the DoF is approximately 6 meters, in our experiments we had much closer shots too. The lenses are also limiting the resolution (especially near the edges of the images), and we went a long way to improve the quality by postprocessing [14]. You may analyze the raw/processed images pairs used in one of these panoramas by our image comparison tool [15] (works in FF better than in Chrome). Currently this postprocessing only corrects aberrations (not the distortions), so the panoramas are stitched manually. We never invested much time into perfecting the panorama stitching, because we have to do it automatically in the end, and we plan to make a camera to be a measurement tool, able to precisely locate the position of each pixel. The software [16] (it is all under GNU GPLv3) can currently measure the position of the calibration pattern (same one as we use for aberrations correction) is already capable of detecting the pattern nodes locations to the +/-0.05 pixels precision, but the overall pixel mapping processing is not finished yet. We'll describe this calibration procedure more on our blog when finished. Levelling the panoramas - yes, it is lousy too. We already have built the hardware for the IMU (inertial measurement unit)integration, that allows logging the high resolution sensor data (we use ADIS-16375) at the full senor rate (300Hz), we also made the hardware for 5Hz (or higher) GPS integration, including the precise timing pulses. Unfortunately, I was not able to finish the FPGA code for the data logger during that our trip, so the panoramas lack that important data and we'll have to try again when ready (it will have odometer/pedometer input also). We are a Free Software/Open Hardware company, and while we can not give away our cameras (and yes, they are much more expensive than a simple panoramic adapter you can use to make brilliant panoramas), our software is all under GNU GPLv3 license, we "release early" (all the software we use internally is maintained at Sourceforge), our camera documentation is also available under Free license. And we are inviting other developers, who have ideas and experience in the panoramic imaging to join this project. [1] http://blog.elphel.com/wp-content/uploads/2011/04/IMG_5655.jpg [2] http://www3.elphel.com/eyesis [3] http://blog.elphel.com/2011/04/panoramas-from-the-backpack/route1_creek/ [4] http://blog.elphel.com/2011/02/eyesis-4pi/ [5] http://blog.elphel.com/wp-content/uploads/2011/04/route2_goblin_valley_panos.jpeg [6] http://blog.elphel.com/wp-content/uploads/2011/04/IMG_5709a_cr_M.jpg [7] http://blog.elphel.com/wp-content/uploads/2011/04/IMG_5738_sm.jpg [8] http://blog.elphel.com/2011/04/panoramas-from-the-backpack/route3_lwhc_panos/ [9] http://blog.elphel.com/wp-content/uploads/2011/04/IMG_5783_cr1.jpg [10] http://community.elphel.com/files/eyesis/webgl-pano-su/geo_tests.html?n=2&mag_filter=linear&min_filter=linear&timer=false&longitude=-110.817458&latitude=38.581867 [11] http://tech.groups.yahoo.com/group/PanoToolsNG/message/49282 [12] http://blog.elphel.com/2011/03/google-api-v3-custom-streetview/ [13] http://blog.elphel.com/2011/02/eyesis-4pi/ [14] http://blog.elphel.com/2010/12/zoom-in-now-enhance-results/ [15] http://postprocess.elphel.com/sebastian2/viewerV3.php?path=image_sets/series_desert [16] http://elphel.git.sourceforge.net/git/gitweb.cgi?p=elphel/ImageJ-Elphel;a=tree

Click here for the active view (resizeable) Description Works in Firefox 4.0 and Chrome. Does NOT work in Firefox 3.6.X. The above sample shows how custom panoramas (in our case: a panorama we shot with our Elphel Eyesis) can be integrated in a custom StreetView. Elphel Eyesis images is the top-left window. The top-right window [...]

Last week our phone system was broken into and we’ve got a phone bill for some five hundred dollars for the calls to Gambia. That expense was not terrible, but still that amount is usually enough for many months of the phone service for our small company – the international phone rates in the VoIP [...]