‎Docker images

← Older revision Revision as of 22:49, 23 December 2019 (33 intermediate revisions by the same user not shown)Line 175: Line 175:    # Test 3: Run a local script (and include a local dir) in contatiner:   # Test 3: Run a local script (and include a local dir) in contatiner:    https://www.tensorflow.org/install/docker   https://www.tensorflow.org/install/docker  +  +  +==Walkthrough for CUDA 10.2 (Dec 2019)==  +  +===Install CUDA===  +* In this [https://www.tensorflow.org/install/gpu guide] there's a [https://developer.nvidia.com/cuda-toolkit-archive link to CUDA toolkit].  +** The toolkit (CUDA Toolkit 10.2) also updated the system driver to 440.33.01  +** Will have to reboot  +  +===Docker===  +====Instructions====  +'''https://www.tensorflow.org/install/docker'''  +  +Quote:  + Docker is the easiest way to enable TensorFlow GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).  +  +====Docker images====  +Where to browse: https://hub.docker.com/r/tensorflow/tensorflow/:  +{| class='wikitable'  +!TF version  +!Python major version  +!GPU support  +!NAME:TAG for Docker command  +|-  +|align='center'|1.15  +|align='center'|3  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:1.15.0-gpu-py3'''  +|-  +|align='center'|2.0.0+  +|align='center'|3  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu-py3'''  +|-  +|align='center'|2.0.0+  +|align='center'|2  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu'''  +|}  +  +====nvidia-docker====  +Somehow it was already installed.  +  +* Check NVIDIA docker version  + ~$ nvidia-docker version  +  +* In the docs it's clear that Docker version 19.03+ should use nvidia-docker2. For Docker of older versions - nvidia-docker v1 should be used.  +* It's not immediately clear about the '''nvidia-container-runtime'''. nvidia-docker v1 & v2 already register it.  +  +====Notes====  +* Can mount a local directory in a 'binding' mode - i.e., update files locally so they are updated in the docker container as well:  + <font size='2'># this will bind-mount directory '''target''' located in '''$(pwd)''', which is a dir the command is run from  + # to '''/app''' in the docker container  +  + ~$ '''docker run \'''  +    '''-it \'''  +    '''--rm \'''  +    '''--name devtest \'''  +    '''-p 0.0.0.0:6006:6006 \'''  +    '''--mount type=bind,source="$(pwd)"/target,target=/app \'''  +    '''--gpus all \'''  +    <font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu-py3</font> \'''  +    '''bash'''</font>  +  +* How to run tensorboard from the container:  + <font size='2'># from [https://briancaffey.github.io/2017/11/20/using-tensorflow-and-tensor-board-with-docker.html here]  + # From the running container's command line (since it was run with 'bash' in the step above):  + root@e9efee9e3fd3:/# '''tensorboard --bind_all --logdir=/app/log.txt'''  # remove --bind_all for TF 1.15  + # Then open a browser:  + '''http://localhost:6006'''</font> Oleg

‎Notes

← Older revision Revision as of 21:56, 23 December 2019 (31 intermediate revisions by the same user not shown)Line 175: Line 175:    # Test 3: Run a local script (and include a local dir) in contatiner:   # Test 3: Run a local script (and include a local dir) in contatiner:    https://www.tensorflow.org/install/docker   https://www.tensorflow.org/install/docker  +  +  +==Walkthrough for CUDA 10.2 (Dec 2019)==  +  +===Install CUDA===  +* In this [https://www.tensorflow.org/install/gpu guide] there's a [https://developer.nvidia.com/cuda-toolkit-archive link to CUDA toolkit].  +** The toolkit (CUDA Toolkit 10.2) also updated the system driver to 440.33.01  +** Will have to reboot  +  +===Docker===  +====Instructions====  +'''https://www.tensorflow.org/install/docker'''  +  +Quote:  + Docker is the easiest way to enable TensorFlow GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).  +  +====Docker images====  +Where to browse: https://hub.docker.com/r/tensorflow/tensorflow/:  +{| class='wikitable'  +!TF version  +!Python major version  +!GPU support  +!TAG for Docker command  +|-  +|align='center'|1.15  +|align='center'|3  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:1.15.0-gpu-py3'''  +|-  +|align='center'|2.0.0+  +|align='center'|3  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu-py3'''  +|-  +|align='center'|2.0.0+  +|align='center'|2  +|align='center'|yes  +|<font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu'''  +|}  +  +====nvidia-docker====  +Somehow it was already installed.  +  +* Check NVIDIA docker version  + ~$ nvidia-docker version  +  +* In the docs it's clear that Docker version 19.03+ should use nvidia-docker2. For Docker of older versions - nvidia-docker v1 should be used.  +* It's not immediately clear about the '''nvidia-container-runtime'''. nvidia-docker v1 & v2 already register it.  +  +====Notes====  +* Can mount a local directory in a 'binding' mode - i.e., update files locally so they are updated in the docker container as well:  + <font size='2'># this will bind-mount directory '''target''' located in '''$(pwd)''', which is a dir the command is run from  + # to '''/app''' in the docker container  +  + ~$ '''docker run \'''  +    '''-it \'''  +    '''--rm \'''  +    '''--name devtest \'''  +    '''-p 0.0.0.0:6006:6006 \'''  +    '''--mount type=bind,source="$(pwd)"/target,target=/app \'''  +    '''--gpus all \'''  +    <font color='darkgreen'>'''tensorflow/tensorflow:latest-gpu-py3</font> \'''  +    '''bash'''</font>  +  +* How to run tensorboard from the container:  + <font size='2'># from [https://briancaffey.github.io/2017/11/20/using-tensorflow-and-tensor-board-with-docker.html here]  + # From the running container's command line (since it was run with 'bash' in the step above):  + root@e9efee9e3fd3:/# '''tensorboard --bind_all --logdir=/app/log.txt'''  + # Then open a browser:  + '''http://localhost:6006'''</font> Oleg

‎Notes

← Older revision Revision as of 21:20, 23 December 2019 (27 intermediate revisions by the same user not shown)Line 175: Line 175:    # Test 3: Run a local script (and include a local dir) in contatiner:   # Test 3: Run a local script (and include a local dir) in contatiner:    https://www.tensorflow.org/install/docker   https://www.tensorflow.org/install/docker  +  +  +==Walkthrough for CUDA 10.2 (Dec 2019)==  +  +===Install CUDA===  +* In this [https://www.tensorflow.org/install/gpu guide] there's a [https://developer.nvidia.com/cuda-toolkit-archive link to CUDA toolkit].  +** The toolkit (CUDA Toolkit 10.2) also updated the system driver to 440.33.01  +** Will have to reboot  +  +===Docker===  +====Instructions====  +'''https://www.tensorflow.org/install/docker'''  +  +Quote:  + Docker is the easiest way to enable TensorFlow GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).  +  +====Docker images====  +Where to browse: https://hub.docker.com/r/tensorflow/tensorflow/  +* tag for python2 + gpu: '''tensorflow/tensorflow:latest-gpu'''  +* tag for python3 + gpu: '''tensorflow/tensorflow:latest-gpu-py3'''  +  +====nvidia-docker====  +Somehow it was already installed.  +  +* Check NVIDIA docker version  + ~$ nvidia-docker version  +  +* In the docs it's clear that Docker version 19.03+ should use nvidia-docker2. For Docker of older versions - nvidia-docker v1 should be used.  +* It's not immediately clear about the '''nvidia-container-runtime'''. nvidia-docker v1 & v2 already register it.  +  +====Notes====  +* Can mount a local directory in a 'binding' mode - i.e., update files locally so they are updated in the docker container as well:  + <font size='2'># this will bind-mount directory '''target''' located in '''$(pwd)''', which is a dir the command is run from  + # to '''/app''' in the docker container  +  + ~$ '''docker run \'''  +    '''-it \'''  +    '''--rm \'''  +    '''--name devtest \'''  +    '''-p 0.0.0.0:6006:6006 \'''  +    '''--mount type=bind,source="$(pwd)"/target,target=/app \'''  +    '''--gpus all \'''  +    '''tensorflow/tensorflow:latest-gpu-py3 \'''  +    '''bash'''</font>  +  +* How to run tensorboard from the container:  + <font size='2'># from [https://briancaffey.github.io/2017/11/20/using-tensorflow-and-tensor-board-with-docker.html here]  + # From the running container's command line (since it was run with 'bash' in the step above):  + root@e9efee9e3fd3:/# '''tensorboard --bind_all --logdir=/app/log.txt'''  + # Then open a browser:  + '''http://localhost:6006'''</font> Oleg

‎Walkthrough for CUDA 10.2 (Dec 2019)

← Older revision Revision as of 20:17, 23 December 2019 (10 intermediate revisions by the same user not shown)Line 175: Line 175:    # Test 3: Run a local script (and include a local dir) in contatiner:   # Test 3: Run a local script (and include a local dir) in contatiner:    https://www.tensorflow.org/install/docker   https://www.tensorflow.org/install/docker  +  +  +==Walkthrough for CUDA 10.2 (Dec 2019)==  +  +===Install CUDA===  +* In this [https://www.tensorflow.org/install/gpu guide] there's a [https://developer.nvidia.com/cuda-toolkit-archive link to CUDA toolkit].  +** The toolkit (CUDA Toolkit 10.2) also updated the system driver to 440.33.01  +** Reboot  +  +===Docker===  +  +====Instructions====  +https://www.tensorflow.org/install/docker  +  +====nvidia-docker====  +* Check NVIDIA docker version  + ~$ nvidia-docker version  +  +In the docs it's clear that Docker version 19.03+ should use nvidia-docker2. For Docker of older versions - nvidia-docker v1 should be used.  +  +It's not immediately clear about the '''nvidia-container-runtime'''. What? Why? Is it automatically installed with nvidia-docker thing?  +  +====Notes====  +* Can mount a local directory in a 'binding' mode - i.e., update files locally so they are updated in the docker container as well:  + <font size='3'># this will bind-mount directory '''target''' located in '''$(pwd)''', which is a dir the command is run from  + # to '''/app''' in the docker container  + docker run \  +    --gpus all \  +    --name sometest \  +    --mount type=bind,source="$(pwd)"/target,target=/app \  +    -it \  +    tensorflow/tensorflow:latest-gpu \  +    bash</font> Oleg

‎Walkthrough for CUDA 10.2 (Dec 2019)

← Older revision Revision as of 19:15, 23 December 2019 (3 intermediate revisions by the same user not shown)Line 175: Line 175:    # Test 3: Run a local script (and include a local dir) in contatiner:   # Test 3: Run a local script (and include a local dir) in contatiner:    https://www.tensorflow.org/install/docker   https://www.tensorflow.org/install/docker  +  +  +==Walkthrough for CUDA 10.2 (Dec 2019)==  +  +===Install CUDA===  +* In this [https://www.tensorflow.org/install/gpu guide] there's a [https://developer.nvidia.com/cuda-toolkit-archive link to CUDA toolkit].  +** The toolkit (CUDA Toolkit 10.2) also updated the system driver to 440.33.01  +** Reboot Oleg

AndreyFilippov committed changes to the Elphel git project :
Started correction of the tiles distortions for lens distortions during correlation

← Older revision Revision as of 22:01, 5 December 2019 Line 9: Line 9:  |[https://arxiv.org/pdf/1911.06975.pdf Filippov, Andrey and Dzhimiev, Oleg "Long Range 3D with Quadocular Thermal (LWIR) Camera" arXiv preprint arXiv:1911.06975 (2019).] |[https://arxiv.org/pdf/1911.06975.pdf Filippov, Andrey and Dzhimiev, Oleg "Long Range 3D with Quadocular Thermal (LWIR) Camera" arXiv preprint arXiv:1911.06975 (2019).]  |Elphel publication |Elphel publication  +|-  +|2019  +|[https://www.researchgate.net/profile/Cui_Xiangbin/publication/335406867_The_conditions_of_the_formation_and_existence_of_Blue_Ice_Areas_in_the_ice_flow_transition_region_from_the_Antarctic_Ice_Sheet_to_the_Amery_Ice_Shelf_in_the_Larsemann_Hills_area/links/5d675ab5299bf11adf29bb92/The-conditions-of-the-formation-and-existence-of-Blue-Ice-Areas-in-the-ice-flow-transition-region-from-the-Antarctic-Ice-Sheet-to-the-Amery-Ice-Shelf-in-the-Larsemann-Hills-area.pdf Markov, Aleksey and Polyakov, Sergey and Sun, Bо and Lukin, Valeriy and Popov, Sergey and Yang, Huigen and Zhang, Tijun and Cui, Xiangbin and Guo, Jingxue and Cui, Penghui and others "Polar Science"]  +|Used Ephel camera in experimental setup  +  |- |-  |2019 |2019  |[https://documat.unirioja.es/descarga/articulo/6802185.pdf Campbell, Andrew, Alan Both, and Qian Chayn Sun. "Detecting and mapping traffic signs from Google Street View images using deep learning and GIS."] |[https://documat.unirioja.es/descarga/articulo/6802185.pdf Campbell, Andrew, Alan Both, and Qian Chayn Sun. "Detecting and mapping traffic signs from Google Street View images using deep learning and GIS."]  |Elphel cameras for GSV referenced   |Elphel cameras for GSV referenced    +  |- |-  |2019 |2019 Andrey.filippov

← Older revision Revision as of 01:12, 27 November 2019 (One intermediate revision by the same user not shown)Line 9: Line 9:  |[https://arxiv.org/pdf/1911.06975.pdf Filippov, Andrey and Dzhimiev, Oleg "Long Range 3D with Quadocular Thermal (LWIR) Camera" arXiv preprint arXiv:1911.06975 (2019).] |[https://arxiv.org/pdf/1911.06975.pdf Filippov, Andrey and Dzhimiev, Oleg "Long Range 3D with Quadocular Thermal (LWIR) Camera" arXiv preprint arXiv:1911.06975 (2019).]  |Elphel publication |Elphel publication  +|-  +|2019  +|[https://documat.unirioja.es/descarga/articulo/6802185.pdf Campbell, Andrew, Alan Both, and Qian Chayn Sun. "Detecting and mapping traffic signs from Google Street View images using deep learning and GIS."]  +|Elphel cameras for GSV referenced  +|-  +|2019  +|[https://documat.unirioja.es/descarga/articulo/6802185.pdf Díaz, Hernán Porras, Duvan Yahir Sanabria Echeverry, and Johan Alexander Ortiz Ferreira. "Tendencia mundial en tecnologías de sistemas de mapeo móvil implementadas con láser."]  +|Elphel Eyesis referenced  |- |-  |2019 |2019 Andrey.filippov

← Older revision Revision as of 21:34, 19 November 2019 (2 intermediate revisions by the same user not shown)Line 5: Line 5:  ! Citation/link ! Citation/link  ! Comments ! Comments  +|-  +|2019  +|[https://arxiv.org/pdf/1911.06975.pdf Filippov, Andrey and Dzhimiev, Oleg "Long Range 3D with Quadocular Thermal (LWIR) Camera" arXiv preprint arXiv:1911.06975 (2019).]  +|Elphel publication  |- |-  |2019 |2019 Line 19: Line 23:  |- |-  |2018 |2018 −|[https://arxiv.org/pdf/1811.08032 Filippov, Andrey, and Oleg Dzhimiev. "See far with TPNET: a Tile Processor and a CNN Symbiosis." arXiv preprint arXiv:1811.08032 (2018).]+|[https://arxiv.org/pdf/1811.08032 Filippov, Andrey, and Dzhimiev, Oleg. "See far with TPNET: a Tile Processor and a CNN Symbiosis." arXiv preprint arXiv:1811.08032 (2018).]  |Elphel publication |Elphel publication  |- |- Andrey.filippov

‎[SOLVED] Note 14: fixdep: Permission denied

← Older revision Revision as of 23:49, 3 October 2019 (6 intermediate revisions by the same user not shown)Line 260: Line 260:    ...   ...    ---[ end Kernel panic - not syncing: Fatal exception in interrupt   ---[ end Kernel panic - not syncing: Fatal exception in interrupt  +  +==<font color='green'>'''[SOLVED]'''</font> Note 14: fixdep: Permission denied==  +* Description:  + - We've had this error for a while, probably since kernel 4.0  + - usually happened when running do_compile_kernelmodules  + - EXTRA_OEMAKE = "-s -w '''-B''' KCFLAGS='-v'"  + - That '''-B''' forces to rebuild all targets and we also have '''-j8''' (in PARALLEL_MAKE variable) for the parallel build  +  - so when running the parallel build fixdep gets rebuilt several times and at some point  +    one of the targets (e.g. sortextable or kallsyms) calls it while fixdep is being compiled and overwritten for another target (probably)  +  - the exec rights are correct after the fact  +  +* Solution:  +  Removed '''-B'''. It make fixdep build only once and the problem is gone.  +  +* Note:  +  ~$ make -h  +    ...  +    -B, --always-make          Unconditionally make all targets.  +    ... Oleg

Oleg Dzhimiev committed changes to the Elphel git project :
updated - ran: scripts/used_file.py linux

Oleg Dzhimiev committed changes to the Elphel git project :
hide comments

Oleg Dzhimiev committed changes to the Elphel git project :
removed fatal message from the init script

Oleg Dzhimiev committed changes to the Elphel git project :
EXTRA_OEMAKE is reassigned a few lines below. Commenting out.

Oleg Dzhimiev committed changes to the Elphel git project :
fixed fixdep permission denied problem by removing '-B'(build all targets unconditionally) option from make

Oleg Dzhimiev committed changes to the Elphel git project :
edit comment

Oleg Dzhimiev committed changes to the Elphel git project :
comment

Oleg Dzhimiev committed changes to the Elphel git project :
added notes to the console while building

Oleg Dzhimiev committed changes to the Elphel git project :
+comment

Oleg Dzhimiev committed changes to the Elphel git project :
fixed bundling after running clean on linux-xlnx