‎A few words on TF in Maven Central repository

Show changes Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (37 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2  −* Modify Tensorflow JNI functions  −* Create a maven project and use modified JNI     −==Install==+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''.   −* In Kubuntu  −* Get TF 1.15.0 - git or archive  −* Install bazel 0.25.2 - see *.deb in releases     −==Build==+How to: − cd ~/git/tensorflow-1.15.0+* '''Build TF JNI''' - libtensorflow.jar, libtensorflow_jni.so and pom.xml − ./configure+* '''Add TF JAR to local Maven''' which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository] − bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+* '''Modify TF JNI functions''' − bazel build -c opt //tensorflow/java:pom+* '''Create Elipse project''' − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml     −Artifacts of interest in bazel-bin/tensorflow/java/:+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. − libtensorflow_jni.so  − libtensorflow.jar  − pom.xml     −* '''xml''' and '''jar''' are taken care of.+==<font color='blue'>Install</font>==  +In Kubuntu:  +* Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0]  +* Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]  +   +Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].  +   +==<font color='blue'>Build</font>==  + <font size=2>cd ~/git/tensorflow-1.15.0  + ./configure # do not forget CUDA  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>  +   +With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  +Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0</font>  +   +Artifacts of interest are in '''bazel-bin/tensorflow/java/''':  + <font size=2>'''libtensorflow_jni.so'''  + '''libtensorflow.jar'''  + '''pom.xml'''</font>  +   +* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Modify TF JNI==+==<font color='blue'>Install JAR to local Maven Repository</font>==  + <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>  +   +How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +   +==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +Rebuild and Reinstall.  +  +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>  +  +==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  +* Create a new maven project  +* Edit pom.xml:  + <font size=2><project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project></font>  +  +===Basic example code===  +tfhello.java:  + <font size=2>import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + }</font>  +  +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +  +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +  +Archive contains Java classes.  +  +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +  +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎Java Maven project in Eclipse

← Older revision Revision as of 01:28, 5 March 2020 (35 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2  −* Modify Tensorflow JNI functions  −* Create a maven project and use modified JNI     −==Install==+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''.   −* In Kubuntu  −* Get TF 1.15.0 - git or archive  −* Install bazel 0.25.2 - see *.deb in releases     −==Build==+How to: − cd ~/git/tensorflow-1.15.0+* '''Build TF JNI''' - libtensorflow.jar, libtensorflow_jni.so and pom.xml − ./configure+* '''Add TF JAR to local Maven''' which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository] − bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+* '''Modify TF JNI functions''' − bazel build -c opt //tensorflow/java:pom+* '''Create Elipse project''' − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml     −Artifacts of interest in bazel-bin/tensorflow/java/:+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. − libtensorflow_jni.so  − libtensorflow.jar  − pom.xml     −* '''xml''' and '''jar''' are taken care of.+==<font color='blue'>Install</font>==  +In Kubuntu:  +* Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0]  +* Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]  +   +Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].  +   +==<font color='blue'>Build</font>==  + <font size=2>cd ~/git/tensorflow-1.15.0  + ./configure # do not forget CUDA  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>  +   +With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  +Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0</font>  +   +Artifacts of interest are in '''bazel-bin/tensorflow/java/''':  + <font size=2>'''libtensorflow_jni.so'''  + '''libtensorflow.jar'''  + '''pom.xml'''</font>  +   +* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Modify TF JNI==+==<font color='blue'>Install JAR to local Maven Repository</font>==  + <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>  +   +How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +   +==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +Rebuild and Reinstall.  +  +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>  +  +==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  +* Create a new maven project  +* Edit pom.xml:  + <font size=2><project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project></font>  +  +===Basic example code===  +tfhello.java:  + <font size=2>import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + }</font> Oleg

‎About

← Older revision Revision as of 00:26, 5 March 2020 (13 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so+  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''. −* Modify Tensorflow JNI functions+  −* Create a maven project and use modified JNI+How to:  +* Build TF JNI - '''libtensorflow.jar, libtensorflow_jni.so'''  +* Add '''libtensorflow.jar''' to local Maven repository which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository]  +* Modify TF JNI functions  +* Create a Maven project and use modified TF JNI  +   +There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. But they fail to clearly explain what it does.     ==Install== ==Install== Line 16: Line 21:  ==Build== ==Build==    cd ~/git/tensorflow-1.15.0   cd ~/git/tensorflow-1.15.0 −  ./configure+  ./configure # do not forget CUDA −  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom − bazel build -c opt //tensorflow/java:pom+  − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml+With tensorflow bazel (when restarted?) tends to rebuild everything from scratch which takes a ton of time. Bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0     Artifacts of interest in bazel-bin/tensorflow/java/: Artifacts of interest in bazel-bin/tensorflow/java/: Line 28: Line 34:  * '''xml''' and '''jar''' are taken care of. * '''xml''' and '''jar''' are taken care of.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"  +  +==Install TF to local maven repo==  + mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml  +  +How to uninstall maven local repo - and switch back to official versions form Central:  + https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install     ==Modify TF JNI== ==Modify TF JNI== Line 38: Line 50:  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +* rebuild and reinstall to mvn  +  +==Java maven project in Eclipse==  +* Create a new maven project  +* Edit pom.xml:  + <project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project>  +  +===Basic example code===  +tfhello.java:  + import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + } Oleg

‎Java maven project in Eclipse

← Older revision Revision as of 23:23, 4 March 2020 (6 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About==  Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this]. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].  +  +And this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md].     How to:   How to:   Line 38: Line 40:  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +* rebuild and reinstall to mvn  +  +==Java maven project in Eclipse==  +* Create a new maven project  +* Edit pom.xml:  + <project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project> Oleg

Created page with "==About== Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this]. How to: * Install things * Build libtensorflow.jar, li..."

New page

==About==
Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].

How to:
* Install things
* Build libtensorflow.jar, libtensorflow_jni.so
* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2
* Modify Tensorflow JNI functions
* Create a maven project and use modified JNI

==Install==
* In Kubuntu
* Get TF 1.15.0 - git or archive
* Install bazel 0.25.2 - see *.deb in releases

==Build==
cd ~/git/tensorflow-1.15.0
./configure
bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni
bazel build -c opt //tensorflow/java:pom
mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml

Artifacts of interest in bazel-bin/tensorflow/java/:
libtensorflow_jni.so
libtensorflow.jar
pom.xml

* '''xml''' and '''jar''' are taken care of.
* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"

==Modify TF JNI==
For example, one wants to create a new function in org.tensorflow.TensorFlow package.
Then, see:
'''tensorflow/java/src/main/java/org/tensorflow/'''
'''tensorflow/java/src/main/native/'''

* add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java
* add to header file tensorflow/java/src/main/native/tensorflow_jni.h
* add to c file tensorflow/java/src/main/native/tensorflow_jni.cc Oleg

‎For Java

← Older revision Revision as of 18:48, 20 February 2020 (One intermediate revision by the same user not shown)Line 113: Line 113:    cd ..   cd ..    ./run_tf_elphel.sh 1</font>   ./run_tf_elphel.sh 1</font>  +  +==For Java==  +* https://www.tensorflow.org/install/lang_java which leads to a more detailed:  +* https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md  +  +In short, to Build TF for Java from source:  + <font size=2>./configure  + bazel build --config opt \  +  //tensorflow/java:tensorflow \  +  //tensorflow/java:libtensorflow_jni</font> Oleg

← Older revision Revision as of 17:04, 15 January 2020 (25 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About==  +Reference guides:  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-from-glacial-to-lightning-speed/ Supercharging Object Detection in Video: from Glacial to Lightning Speed]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-videos-setup/ Setup]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-first-app/ First App]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-optimizing-decoding-and-graph-feeding/ Optimizing Decoding and Graph Feeding]  +  +Target projects:  +* '''tensorflow-object-detection-cpp''' - contains a sample model, labels and a test video  +* '''fast_od''' - contains feeding Tensorflow from GPU     ==Setup== ==Setup==  +Kubuntu 18.04, CUDA 10.0 (10.0 is a requirement for Java version of TF 1.15 in Maven CDN):  + <font size=2>sudo apt install build-essential  + sudo apt install git curl  +  + mkdir -p ~/git/tf_cv_cu  + cd ~/git/tf_cv_cu  +  + '''# Download section'''  + <nowiki>git clone https://github.com/fierval/tensorflow-object-detection-cpp.git  +git clone https://github.com/fierval/fast_od.git  +wget https://github.com/Kitware/CMake/releases/download/v3.16.2/cmake-3.16.2.tar.gz  +wget https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb  +wget https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz -O tensorflow-1.15.0.tar.gz  +wget https://github.com/opencv/opencv/archive/3.4.9.tar.gz -O opencv-3.4.9.tar.gz  +wget https://github.com/opencv/opencv_contrib/archive/3.4.9.tar.gz -O opencv_contrib-3.4.9.tar.gz</nowiki>  +  + '''# Unpack archives'''  + tar xzvf cmake-3.16.2.tar.gz  + tar xzvf tensorflow-1.15.0.tar.gz  + tar xzvf opencv-3.4.9.tar.gz  + tar xzvf opencv_contrib-3.4.9.tar.gz  +  + '''# Build & update cmake'''  + sudo apt install libssl-dev  + sudo apt install qt4-default  + cd cmake-3.16.2  + ./bootstrap -qt-gui  + make -j8  + sudo make install  +  + cmake -version  + cmake-gui -version  +  + '''# Install bazel'''  + sudo dpkg -i bazel_0.25.2-linux-x86_64.deb  + bazel version  +  + '''# Build TF'''  + cd ~/git/tf_cv_cu/tensorflow-1.15.0/  + tensorflow/contrib/makefile/download_dependencies.sh  + ./configure  + '''# pick all defaults except select '''yes''' for CUDA support'''  + bazel build //tensorflow:libtensorflow_cc.so  +  + '''# Install TF with headers'''  + sudo mkdir /usr/local/tensorflow  + sudo mkdir /usr/local/tensorflow/include  + sudo cp -r tensorflow/contrib/makefile/downloads/eigen/Eigen /usr/local/tensorflow/include/  + sudo cp -r tensorflow/contrib/makefile/downloads/eigen/unsupported /usr/local/tensorflow/include/  + sudo cp tensorflow/contrib/makefile/downloads/nsync/public/* /usr/local/tensorflow/include/  + sudo cp -r bazel-genfiles/tensorflow /usr/local/tensorflow/include/  + sudo cp -r tensorflow/cc /usr/local/tensorflow/include/tensorflow  + sudo cp -r tensorflow/core /usr/local/tensorflow/include/tensorflow  + sudo mkdir /usr/local/tensorflow/include/third_party  + sudo cp -r third_party/eigen3 /usr/local/tensorflow/include/third_party/  + sudo mkdir /usr/local/tensorflow/lib  + sudo cp bazel-bin/tensorflow/libtensorflow_*.so /usr/local/tensorflow/lib  + '''# extras, not mentioned in the reference guide'''  + sudo cp -r tensorflow/contrib/makefile/downloads/absl/absl /usr/local/tensorflow/include/  + sudo cp -r tensorflow/contrib/makefile/downloads/protobuf/src/google /usr/local/tensorflow/include/  + sudo cp -r tensorflow/stream_executor /usr/local/tensorflow/include/tensorflow/  + sudo cp bazel-bin/tensorflow/* /usr/local/tensorflow/lib   + sudo ln -sf /usr/local/tensorflow/lib/libtensorflow_framework.so.1.15.0 /usr/local/tensorflow/lib/libtensorflow_framework.so  +  + '''# Build & install OpenCV'''  + sudo apt install libgtkglext1 libgtkglext1-dev  + sudo apt install libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev  + sudo apt install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev  + # sudo apt install libjasper-dev '''# no such package in 18.04'''  +  + sudo cp ~/git/tf_cv_cu/fast_od/docker/nvcuvid/dynlink_* /usr/local/cuda/include  +  + cd ~/git/tf_cv_cu/opencv-3.4.9  + mkdir build; cd build  + '''# haven't tested this line yet. Did from cmake-gui'''  + cmake -D WITH_CUDA=ON \  +      -D OPENCV_EXTRA_MODULES_PATH=~/git/tf_cv_cu/opencv_contrib-3.4.9/modules \  +      -D BUILD_PERF_TESTS=OFF \  +      -D BUILD_TESTS=OFF \  +      -D BUILD_opencv_cudacodec=ON \  +      -D WITH_NVCUVID=ON \  +      -D WITH_GTK_2_X=ON \  +      ..  + make -j8  + sudo make install  + sudo ldconfig  +  + '''# tensorflow-object-detection-cpp'''  + cd ~/git/tf_cv_cu/tensorflow-object-detection-cpp  + mkdir build; cd build  + cmake ..  + make  + ./tf_detector_example  +  + '''# fast_od'''  + cd ~/git/tf_cv_cu/fast_od  + mkdir build; cd build  + cmake ..  + make  + '''# update paths paths to the model, labels and the video from tensorflow-object-detection-cpp in fast_od/run_tf.sh'''  + cd ..  + ./run_tf_elphel.sh 1</font> Oleg

‎Setup

← Older revision Revision as of 01:56, 15 January 2020 (23 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About==  +Reference guides:  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-from-glacial-to-lightning-speed/ Supercharging Object Detection in Video: from Glacial to Lightning Speed]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-videos-setup/ Setup]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-first-app/ First App]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-optimizing-decoding-and-graph-feeding/ Optimizing Decoding and Graph Feeding]  +  +Target projects:  +* '''tensorflow-object-detection-cpp''' - contains a sample model, labels and a test video  +* '''fast_od''' - contains feeding Tensorflow from GPU     ==Setup== ==Setup==  +Kubuntu 18.04, CUDA 10.0 (10.0 is a requirement for Java version of TF 1.15 in Maven CDN):  + <font size=2>sudo apt install build-essential  + sudo apt install git curl  +  + mkdir -p ~/git/tf_cv_cu  + cd ~/git/tf_cv_cu  +  + '''# Download section'''  + <nowiki>git clone https://github.com/fierval/tensorflow-object-detection-cpp.git  +git clone https://github.com/fierval/fast_od.git  +wget https://github.com/Kitware/CMake/releases/download/v3.16.2/cmake-3.16.2.tar.gz  +wget https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb  +wget https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz -O tensorflow-1.15.0.tar.gz  +wget https://github.com/opencv/opencv/archive/3.4.9.tar.gz -O opencv-3.4.9.tar.gz  +wget https://github.com/opencv/opencv_contrib/archive/3.4.9.tar.gz -O opencv_contrib-3.4.9.tar.gz</nowiki>  +  + '''# Unpack archives'''  + tar xzvf cmake-3.16.2.tar.gz  + tar xzvf tensorflow-1.15.0.tar.gz  + tar xzvf opencv-3.4.9.tar.gz  + tar xzvf opencv_contrib-3.4.9.tar.gz  +  + '''# Build & update cmake'''  + sudo apt install libssl-dev  + sudo apt install qt4-default  + cd cmake-3.16.2  + ./bootstrap -qt-gui  + make -j8  + sudo make install  +  + cmake -version  + cmake-gui -version  +  + '''# Install bazel'''  + sudo dpkg -i bazel_0.25.2-linux-x86_64.deb  + bazel version  +  + '''# Build TF'''  + cd ~/git/tf_cv_cu/tensorflow-1.15.0/  + tensorflow/contrib/makefile/download_dependencies.sh  + ./configure  + '''# pick all defaults except select '''yes''' for CUDA support'''  + bazel build //tensorflow:libtensorflow_cc.so  +  + '''# Install TF with headers'''  + sudo mkdir /usr/local/tensorflow  + sudo mkdir /usr/local/tensorflow/include  + sudo cp -r tensorflow/contrib/makefile/downloads/eigen/Eigen /usr/local/tensorflow/include/  + sudo cp -r tensorflow/contrib/makefile/downloads/eigen/unsupported /usr/local/tensorflow/include/  + sudo cp tensorflow/contrib/makefile/downloads/nsync/public/* /usr/local/tensorflow/include/  + sudo cp -r bazel-genfiles/tensorflow /usr/local/tensorflow/include/  + sudo cp -r tensorflow/cc /usr/local/tensorflow/include/tensorflow  + sudo cp -r tensorflow/core /usr/local/tensorflow/include/tensorflow  + sudo mkdir /usr/local/tensorflow/include/third_party  + sudo cp -r third_party/eigen3 /usr/local/tensorflow/include/third_party/  + sudo mkdir /usr/local/tensorflow/lib  + sudo cp bazel-bin/tensorflow/libtensorflow_*.so /usr/local/tensorflow/lib  + '''# extras, not mentioned in the reference guide'''  + sudo cp -r tensorflow/contrib/makefile/downloads/absl/absl /usr/local/tensorflow/include/  + sudo cp -r tensorflow/contrib/makefile/downloads/protobuf/src/google /usr/local/tensorflow/include/  + sudo cp -r tensorflow/stream_executor /usr/local/tensorflow/include/tensorflow/  + sudo cp bazel-bin/tensorflow/* /usr/local/tensorflow/lib   + sudo ln -sf /usr/local/tensorflow/lib/libtensorflow_framework.so.1.15.0 /usr/local/tensorflow/lib/libtensorflow_framework.so  +  + '''# Build & install OpenCV'''  + sudo apt install libgtkglext1 libgtkglext1-dev  + sudo apt install libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev  + sudo apt install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev  + # sudo apt install libjasper-dev '''# no such package in 18.04'''  +  + sudo cp ~/git/tf_cv_cu/fast_od/docker/nvcuvid/dynlink_* /usr/local/cuda/include  +  + cd ~/git/tf_cv_cu/opencv-3.4.9  + mkdir build; cd build  + '''# haven't tested this line yet. Did from cmake-gui'''  + cmake -D WITH_CUDA=ON \  +      -D OPENCV_EXTRA_MODULES_PATH=~/git/tf_cv_cu/opencv_contrib-3.4.9/modules \  +      -D BUILD_PERF_TESTS=OFF \  +      -D BUILD_TESTS=OFF \  +      -D BUILD_opencv_cudacodec=ON \  +      -D WITH_NVCUVID=ON \  +      -D WITH_GTK_2_X=ON \  +      ..  + make -j8  + sudo make install  + sudo ldconfig  + </font> Oleg

‎Setup

← Older revision Revision as of 01:42, 15 January 2020 (16 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About==  +Reference guides:  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-from-glacial-to-lightning-speed/ Supercharging Object Detection in Video: from Glacial to Lightning Speed]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-videos-setup/ Setup]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-first-app/ First App]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-optimizing-decoding-and-graph-feeding/ Optimizing Decoding and Graph Feeding]  +  +Target projects:  +* '''tensorflow-object-detection-cpp''' - contains a sample model, labels and a test video  +* '''fast_od''' - contains feeding Tensorflow from GPU     ==Setup== ==Setup==  +Kubuntu 18.04, CUDA 10.0 (10.0 is a requirement for Java version of TF 1.15 in Maven CDN):  + <font size=2>mkdir -p ~/git/tf_cv_cu  + cd ~/git/tf_cv_cu  +  + '''# Download section'''  + <nowiki>git clone https://github.com/fierval/tensorflow-object-detection-cpp.git  +git clone https://github.com/fierval/fast_od.git  +wget https://github.com/Kitware/CMake/releases/download/v3.16.2/cmake-3.16.2.tar.gz  +wget https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb  +wget https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz -O tensorflow-1.15.0.tar.gz  +wget https://github.com/opencv/opencv/archive/3.4.9.tar.gz -O opencv-3.4.9.tar.gz  +wget https://github.com/opencv/opencv_contrib/archive/3.4.9.tar.gz -O opencv_contrib-3.4.9.tar.gz</nowiki>  +  + '''# Unpack archives'''  + tar xzvf cmake-3.16.2.tar.gz  + tar xzvf tensorflow-1.15.0.tar.gz  + tar xzvf opencv-3.4.9.tar.gz  + tar xzvf opencv_contrib-3.4.9.tar.gz  +  + '''# Build & update cmake'''  + sudo apt install libssl-dev  + sudo apt install qt4-default  + cd cmake-3.16.2  + ./bootstrap -qt-gui  + make -j8  + sudo make install  +  + cmake -version  + cmake-gui -version  +  + '''# Install bazel'''  + sudo dpkg -i bazel_0.25.2-linux-x86_64.deb  + bazel version  +  + '''# Build TF'''  + cd ~/git/tf_cv_cu/tensorflow-1.15.0/  + tensorflow/contrib/makefile/download_dependencies.sh  + ./configure  + # pick all defaults except select '''yes''' for CUDA support  + bazel build //tensorflow:libtensorflow_cc.so  +  + '''# Build OpenCV'''  + sudo apt install libgtkglext1 libgtkglext1-dev  + sudo apt install build-essential  + sudo apt install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev  + sudo apt install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev  + # sudo apt install libjasper-dev # missing in Kubuntu 18.04  +  + sudo cp ~/git/tf_cv_cu/fast_od/docker/nvcuvid/dynlink_* /usr/local/cuda/include  +  + cd ~/git/tf_cv_cu/opencv-3.4.9  + mkdir build; cd build  + cmake -D WITH_CUDA=ON \  +      -D OPENCV_EXTRA_MODULES_PATH=~/git/tf_cv_cu/opencv_contrib-3.4.9/modules \  +      -D BUILD_PERF_TESTS=OFF \  +      -D BUILD_TESTS=OFF \  +      -D BUILD_opencv_cudacodec=ON \  +      -D WITH_NVCUVID=ON \  +      -D WITH_GTK_2_X=ON \  +      ..  + make -j8  + sudo make install  + sudo ldconfig  + </font> Oleg

← Older revision Revision as of 00:30, 15 January 2020 Line 1: Line 1:  ==About== ==About==  +Reference guides:  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-from-glacial-to-lightning-speed/ Supercharging Object Detection in Video: from Glacial to Lightning Speed]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-videos-setup/ Setup]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-first-app/ First App]  +* [https://viralfsharp.com/2019/03/25/supercharging-object-detection-in-video-optimizing-decoding-and-graph-feeding/ Optimizing Decoding and Graph Feeding]  +     ==Setup== ==Setup== Oleg

Created page with "==About== ==Setup=="

New page

==About==

==Setup== Oleg

‎Tensorflow and OpenCV building notes

← Older revision Revision as of 00:49, 8 January 2020 Line 249: Line 249:     ==Tensorflow and OpenCV building notes== ==Tensorflow and OpenCV building notes== −===Targets 1===+===Build 1===  # TF 1.15.0 # TF 1.15.0  # CUDA 10.0 and Toolkit and stuff # CUDA 10.0 and Toolkit and stuff Line 261: Line 261:    3. ./configure   3. ./configure    4.   4.  +  +===Build 2===  +# TF 1.13.1  +# CUDA 10.0 and Toolkit and stuff  +# OpenCV 3.4.9  +  +====TF 1.13.1====  +* Will build with Bazel 0.21.0 (installed from [https://github.com/bazelbuild/bazel/releases/tag/0.21.0 deb archive]) Oleg

Andrey.filippov changed visibility of 6 revisions on page File:333 hd setup.jpg: content hidden, edit summary hidden and username hidden vandalism

Andrey.filippov

‎Notes

← Older revision Revision as of 23:12, 7 January 2020 (One intermediate revision by the same user not shown)Line 1: Line 1: −==Requirements==+==OS==  * Kubuntu 16.04 LTS * Kubuntu 16.04 LTS  +  ==Setup (guide)== ==Setup (guide)==  Just follow: Just follow: Line 246: Line 247:    # Then open a browser:   # Then open a browser:    '''http://localhost:6006'''</font>   '''http://localhost:6006'''</font>  +  +==Tensorflow and OpenCV building notes==  +===Targets 1===  +# TF 1.15.0  +# CUDA 10.0 and Toolkit and stuff  +# OpenCV 3.4.9  +  +====TF 1.15.0====  +* Will build with Bazel 0.25.2 (installed from [https://github.com/bazelbuild/bazel/releases/tag/0.25.2 deb archive])  +* TF - downloaded as [https://github.com/tensorflow/tensorflow/releases/tag/v1.15.0 tensorflow-1.15.0.tar.gz]  + 1. Unpack  + 2. cd tensorflow-1.15.0  + 3. ./configure  + 4. Oleg

‎Requirements

← Older revision Revision as of 20:16, 7 January 2020 Line 1: Line 1: −==Requirements==+==OS==  * Kubuntu 16.04 LTS * Kubuntu 16.04 LTS  +  ==Setup (guide)== ==Setup (guide)==  Just follow: Just follow: Oleg

← Older revision Revision as of 23:19, 27 December 2019 (12 intermediate revisions by the same user not shown)Line 2: Line 2:  * Install Eclipse * Install Eclipse  * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] −<font color='tomato'>'''Note: if the project is updated/pulled outside Eclipse - need manual refresh'''</font>+<font color='red'>'''NOTE: if project is updated/pulled outside Eclipse - might need a manual refresh'''</font>  +* TF version is pulled from pom.xml  +* Trained TF model for EO sensors is auto-downloaded - [https://community.elphel.com/files/quad-stereo/ml/trained_model_v1.0.zip trained_model_v1.0.zip]  +* Get some image samples, provide paths  +* Before running the plugin (Eyesis_Correction), copy imagej options to /home/user/.imagejs/Eyesis_Correction.xml:  +<font size='1'>  + <?xml version="1.0" encoding="UTF-8"?>  + <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">  + <properties>  +    <comment>last updated Thu Sep 08 14:09:47 MDT 2042</comment>  +    <entry key="ADVANCED_MODE">True</entry>  +    <entry key="DCT_MODE">True</entry>  +    <entry key="MODE_3D">False</entry>  +    <entry key="GPU_MODE">True</entry>  +    <entry key="LWIR_MODE">True</entry>  + </properties>  +</font>  +* Updated pom.xml to TF 1.15 - package exists  +* Install cuDNN all 3 packages - runtime, dev and docs. Used docs to verify installation - built mnistCUDNN:  + https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html  +* I think TF 1.15 maven package was built for CUDA 10.0 driver, and so it whines when 10.2 is installed.  + <font size=1 color=red>2019-12-27 13:05:15.754656: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754860: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754970: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755075: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755178: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.762197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 13:05:15.762227: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.  + Skipping registering GPU devices...</font>  +   +* TF 1.15 and CUDA 10.0 require GPU compute capability = 6.0, GeForce GTX 750Ti is 5.0:  + <font color=red size=1>2019-12-27 14:22:17.475717: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/oleg/GIT/imagej-elphel/target/classes/trained_model  + 2019-12-27 14:22:17.477009: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }  + 2019-12-27 14:22:17.503393: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3392030000 Hz  + 2019-12-27 14:22:17.504196: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f610dba1a20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:  + 2019-12-27 14:22:17.504235: I tensorflow/compiler/xla/service/service.cc:176]  StreamExecutor device (0): Host, Default Version  + 2019-12-27 14:22:17.505378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1  + 2019-12-27 14:22:17.517647: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:17.518168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:  + name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.1105  + pciBusID: 0000:01:00.0  + 2019-12-27 14:22:17.518385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0  + 2019-12-27 14:22:17.519624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0  + 2019-12-27 14:22:17.675574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0  + 2019-12-27 14:22:17.716621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0  + 2019-12-27 14:22:18.160070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0  + 2019-12-27 14:22:18.439862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0  + 2019-12-27 14:22:18.443378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 14:22:18.443483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:18.444034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:18.444510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Ignoring visible gpu device (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0) with  + '''Cuda compute capability 5.0. The minimum required Cuda capability is 6.0.'''  + 2019-12-27 14:22:18.498425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:  + 2019-12-27 14:22:18.498455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0  + 2019-12-27 14:22:18.498463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:  N  + 2019-12-27 14:22:18.504855: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.  + 2019-12-27 14:22:18.528948: I tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: /home/oleg/GIT/imagej-elphel/target/classes/trained_model  + 2019-12-27 14:22:18.581034: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 1105321 microseconds.  +</font>  +* So, <font color='darkgreen'>'''TF1.15 + CUDA 10.0 might work with GeForce GTX 1080 Ti (compute capability 6.1)'''</font>  +TF Test button in Eyesis_Correction plugin worked with CUDA 10.0 (even with nvidia-smi showing CUDA 10.1 - it's probably not relevant to the libs used) Oleg

← Older revision Revision as of 21:55, 27 December 2019 (10 intermediate revisions by the same user not shown)Line 2: Line 2:  * Install Eclipse * Install Eclipse  * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] −<font color='tomato'>'''Note: if the project is updated/pulled outside Eclipse - need manual refresh'''</font>+<font color='red'>'''NOTE: if project is updated/pulled outside Eclipse - might need a manual refresh'''</font>  +* TF version is pulled from pom.xml  +* Trained TF model for EO sensors is auto-downloaded - [https://community.elphel.com/files/quad-stereo/ml/trained_model_v1.0.zip trained_model_v1.0.zip]  +* Get some image samples, provide paths  +* Before running the plugin (Eyesis_Correction), copy imagej options to /home/user/.imagejs/Eyesis_Correction.xml:  +<font size='1'>  + <?xml version="1.0" encoding="UTF-8"?>  + <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">  + <properties>  +    <comment>last updated Thu Sep 08 14:09:47 MDT 2042</comment>  +    <entry key="ADVANCED_MODE">True</entry>  +    <entry key="DCT_MODE">True</entry>  +    <entry key="MODE_3D">False</entry>  +    <entry key="GPU_MODE">True</entry>  +    <entry key="LWIR_MODE">True</entry>  + </properties>  +</font>  +* Updated pom.xml to TF 1.15 - package exists  +* Install cuDNN all 3 packages - runtime, dev and docs. Used docs to verify installation - built mnistCUDNN:  + https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html  +* I think TF 1.15 maven package was built for CUDA 10.0 driver, and so it whines when 10.2 is installed.  + <font size=1 color=red>2019-12-27 13:05:15.754656: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754860: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754970: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755075: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755178: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.762197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 13:05:15.762227: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.  + Skipping registering GPU devices...</font>  +   +* TF 1.15 and CUDA 10.0 require GPU compute capability = 6.0, GeForce GTX 750Ti is 5.0:  + <font color=red size=1>2019-12-27 14:22:17.475717: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/oleg/GIT/imagej-elphel/target/classes/trained_model  + 2019-12-27 14:22:17.477009: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }  + 2019-12-27 14:22:17.503393: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3392030000 Hz  + 2019-12-27 14:22:17.504196: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f610dba1a20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:  + 2019-12-27 14:22:17.504235: I tensorflow/compiler/xla/service/service.cc:176]  StreamExecutor device (0): Host, Default Version  + 2019-12-27 14:22:17.505378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1  + 2019-12-27 14:22:17.517647: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:17.518168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:  + name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.1105  + pciBusID: 0000:01:00.0  + 2019-12-27 14:22:17.518385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0  + 2019-12-27 14:22:17.519624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0  + 2019-12-27 14:22:17.675574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0  + 2019-12-27 14:22:17.716621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0  + 2019-12-27 14:22:18.160070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0  + 2019-12-27 14:22:18.439862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0  + 2019-12-27 14:22:18.443378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 14:22:18.443483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:18.444034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero  + 2019-12-27 14:22:18.444510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Ignoring visible gpu device (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0) with  + '''Cuda compute capability 5.0. The minimum required Cuda capability is 6.0.'''  + 2019-12-27 14:22:18.498425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:  + 2019-12-27 14:22:18.498455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0  + 2019-12-27 14:22:18.498463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:  N  + 2019-12-27 14:22:18.504855: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.  + 2019-12-27 14:22:18.528948: I tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: /home/oleg/GIT/imagej-elphel/target/classes/trained_model  + 2019-12-27 14:22:18.581034: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 1105321 microseconds.  +</font>  +* So, <font color='darkgreen'>'''TF1.15 + CUDA 10.0 might work with GeForce GTX 1080 Ti (compute capability 6.1)'''</font> Oleg

← Older revision Revision as of 20:32, 27 December 2019 (7 intermediate revisions by the same user not shown)Line 2: Line 2:  * Install Eclipse * Install Eclipse  * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] −<font color='tomato'>'''Note: if the project is updated/pulled outside Eclipse - need manual refresh'''</font>+<font color='red'>'''NOTE: if project is updated/pulled outside Eclipse - might need a manual refresh'''</font>  +* TF version is pulled from pom.xml  +* Trained TF model for EO sensors is auto-downloaded - [https://community.elphel.com/files/quad-stereo/ml/trained_model_v1.0.zip trained_model_v1.0.zip]  +* Get some image samples, provide paths  +* Before running the plugin (Eyesis_Correction), copy imagej options to /home/user/.imagejs/Eyesis_Correction.xml:  +<font size='1'>  + <?xml version="1.0" encoding="UTF-8"?>  + <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">  + <properties>  +    <comment>last updated Thu Sep 08 14:09:47 MDT 2042</comment>  +    <entry key="ADVANCED_MODE">True</entry>  +    <entry key="DCT_MODE">True</entry>  +    <entry key="MODE_3D">False</entry>  +    <entry key="GPU_MODE">True</entry>  +    <entry key="LWIR_MODE">True</entry>  + </properties>  +</font>  +* Updaed pom.xml to TF 1.15 - package exists  +* Install cuDNN all 3 packages - runtime, dev and docs. Used docs to verify installation - built mnistCUDNN:  + https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html  +* I think TF 1.15 maven package was built for CUDA 10.0 driver, and so it whines when 10.2 is installed.  + <font size=1 color=red>2019-12-27 13:05:15.754656: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754860: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754970: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755075: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755178: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.762197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 13:05:15.762227: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.  + Skipping registering GPU devices...</font> Oleg

← Older revision Revision as of 20:11, 27 December 2019 (6 intermediate revisions by the same user not shown)Line 2: Line 2:  * Install Eclipse * Install Eclipse  * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] * Clone and Import [https://git.elphel.com/Elphel/imagej-elphel imagej-elphel] −<font color='tomato'>'''Note: if the project is updated/pulled outside Eclipse - need manual refresh'''</font>+<font color='red'>'''NOTE: if project is updated/pulled outside Eclipse - might need a manual refresh'''</font>  +* TF version is pulled from pom.xml  +* Trained TF model for EO sensors is auto-downloaded - [https://community.elphel.com/files/quad-stereo/ml/trained_model_v1.0.zip trained_model_v1.0.zip]  +* Get some image samples, provide paths  +* Before running the plugin (Eyesis_Correction), copy imagej options to /home/user/.imagejs/Eyesis_Correction.xml:  +<font size='1'>  + <?xml version="1.0" encoding="UTF-8"?>  + <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">  + <properties>  +    <comment>last updated Thu Sep 08 14:09:47 MDT 2042</comment>  +    <entry key="ADVANCED_MODE">True</entry>  +    <entry key="DCT_MODE">True</entry>  +    <entry key="MODE_3D">False</entry>  +    <entry key="GPU_MODE">True</entry>  +    <entry key="LWIR_MODE">True</entry>  + </properties>  +</font>  +* Updaed pom.xml to TF 1.15 - package exists  +* Install cuDNN all 3 packages - runtime, dev and docs. Used docs to verify installation - built mnistCUDNN:  + https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html  +* I think TF 1.15 maven package was built for CUDA 10.0 driver  + <font size=1 color=red>2019-12-27 13:05:15.754656: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754860: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.754970: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755075: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.755178: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory  + 2019-12-27 13:05:15.762197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7  + 2019-12-27 13:05:15.762227: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.  + Skipping registering GPU devices...</font> Oleg