← Older revision Revision as of 23:36, 17 March 2020 Line 1: Line 1:  +{{Legacy}}  +'''Elphel currently uses git repository at https://git.elphel.com, mirrored at https://github.com/Elphel'''  +  +  = Releases description on Elphel's SourceForge project= = Releases description on Elphel's SourceForge project=    Andrey.filippov

‎Build

← Older revision Revision as of 23:43, 13 March 2020 (4 intermediate revisions by the same user not shown)Line 28: Line 28:     ==<font color='blue'>Build</font>== ==<font color='blue'>Build</font>==  +===Quick===  +~/git/tensorflow-1.15.0/mvn_build.sh:  + <font size=2># Step 1 (Java):  +  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom  + mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml  +  + # Step 2 (JNI):  +  + bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz  +  + rm -rf bazel-bin/tensorflow/tools/lib_package/maven  + mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  +  + POM="  + <project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project>  + "  +  + echo $POM > bazel-bin/tensorflow/tools/lib_package/maven/pom.xml  +  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +  +===Detailed===  +    <font size=2>cd ~/git/tensorflow-1.15.0   <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA Line 76: Line 121:  Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml. Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  Next: Next: −  <font size=2>mvn package+  <font size=2>cd bazel-bin/tensorflow/tools/lib_package/maven −  mvn install</font>+ mvn package  +  mvn install  + cd ../../../../..  +</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Oleg

‎Build

← Older revision Revision as of 23:43, 13 March 2020 (5 intermediate revisions by the same user not shown)Line 18: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 28: Line 28:     ==<font color='blue'>Build</font>== ==<font color='blue'>Build</font>==  +===Quick===  +~/git/tensorflow-1.15.0/mvn_build.sh:  + <font size=2># Step 1 (Java):  +  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom  + mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml  +  + # Step 2 (JNI):  +  + bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz  +  + rm -rf bazel-bin/tensorflow/tools/lib_package/maven  + mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  +  + POM="  + <project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project>  + "  +  + echo $POM > bazel-bin/tensorflow/tools/lib_package/maven/pom.xml  +  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +  +===Detailed===  +    <font size=2>cd ~/git/tensorflow-1.15.0   <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA Line 76: Line 121:  Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml. Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  Next: Next: −  <font size=2>mvn package+  <font size=2>cd bazel-bin/tensorflow/tools/lib_package/maven −  mvn install</font>+ mvn package  +  mvn install  + cd ../../../../..  +</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Oleg

‎Build

← Older revision Revision as of 23:43, 13 March 2020 (19 intermediate revisions by the same user not shown)Line 18: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 28: Line 28:     ==<font color='blue'>Build</font>== ==<font color='blue'>Build</font>==  +===Quick===  +~/git/tensorflow-1.15.0/mvn_build.sh:  + <font size=2># Step 1 (Java):  +  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom  + mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml  +  + # Step 2 (JNI):  +  + bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz  +  + rm -rf bazel-bin/tensorflow/tools/lib_package/maven  + mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  +  + POM="  + <project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project>  + "  +  + echo $POM > bazel-bin/tensorflow/tools/lib_package/maven/pom.xml  +  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +  +===Detailed===  +    <font size=2>cd ~/git/tensorflow-1.15.0   <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA Line 43: Line 88:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===    <font size=2># /usr/lib is in the default java.library.path   <font size=2># /usr/lib is in the default java.library.path    sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>   sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +  +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 52: Line 132:  How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''    −<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>+<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Oleg

← Older revision Revision as of 21:44, 13 March 2020 (18 intermediate revisions by the same user not shown)Line 18: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 43: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===    <font size=2># /usr/lib is in the default java.library.path   <font size=2># /usr/lib is in the default java.library.path    sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>   sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +  +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 52: Line 87:  How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''    −<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>+<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Oleg

‎preferred [option 2] Build so package

← Older revision Revision as of 21:33, 13 March 2020 (16 intermediate revisions by the same user not shown)Line 18: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 43: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===    <font size=2># /usr/lib is in the default java.library.path   <font size=2># /usr/lib is in the default java.library.path    sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>   sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +  +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>cd bazel-bin/tensorflow/tools/lib_package/maven  + mvn package  + mvn install  + cd ../../../../..  +</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 52: Line 87:  How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''    −<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>+<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Oleg

‎About

← Older revision Revision as of 23:57, 10 March 2020 (14 intermediate revisions by the same user not shown)Line 18: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 43: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===    <font size=2># /usr/lib is in the default java.library.path   <font size=2># /usr/lib is in the default java.library.path    sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>   sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +  +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>mvn package  + mvn install</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 52: Line 84:  How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''    −<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>+<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Oleg

‎About

← Older revision Revision as of 23:57, 10 March 2020 (31 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 12: Line 18:  * '''Create Elipse project''' * '''Create Elipse project'''    −There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. <s>Seems useless</s>. Seems useful.     ==<font color='blue'>Install</font>== ==<font color='blue'>Install</font>== Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +   +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>mvn package  + mvn install</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 83:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 119:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 135:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 145:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎preferred [option 2] Build so package

← Older revision Revision as of 23:30, 9 March 2020 (30 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +   +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven/org/tensorflow/native/linux-x86_64</font>  +Next create a pom.xml in ''bazel-bin/tensorflow/tools/lib_package/maven'':  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>mvn package  + mvn install</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 83:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 119:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 135:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 145:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎preferred [option 2] Build so package

← Older revision Revision as of 21:33, 9 March 2020 (29 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...".  +   +===[option 1] Link so library===  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +   +=== preferred [option 2] Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +It puts all libs into a single archive.  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven</font>  +Next create a pom.xml there (''bazel-bin/tensorflow/tools/lib_package/maven''):  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>'''libtensorflow_jni_gpu'''</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Note: '''libtensorflow_jni_gpu''' - name can be any - just make sure you use it in your project's pom.xml.  +Next:  + <font size=2>mvn package  + mvn install</font>  +   +I'm yet to test this but the '''libtensorflow_jni_gpu-1.15.0.jar''' at least has all the files needed.     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 85:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked (or '''jar''''d) and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 121:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 137:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 147:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎Build

← Older revision Revision as of 20:54, 9 March 2020 (20 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path (alternatively see [[Tensorflow_JNI_development#Build_so_package|'''Build so package''']] a little below and skip this linking). Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +   +===Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>  +Now to create a JAR to replace [https://mvnrepository.com/artifact/org.tensorflow/libtensorflow_jni_gpu libtensorflow_jni_gpu], do this:  + <font size=2>mkdir -p bazel-bin/tensorflow/tools/lib_package/maven  + tar -zxvf bazel-bin/tensorflow/tools/lib_package/libtensorflow_jni.tar.gz -C bazel-bin/tensorflow/tools/lib_package/maven</font>  +Next create a pom.xml there:  + <font size=2><project>  +  <modelVersion>4.0.0</modelVersion>  +  <description>Platform-dependent native code for the TensorFlow Java library. CUDA support depends on the local build.</description>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.15.0</version>  +  <packaging>jar</packaging>  +  <build>  +    <resources>  +      <resource>  +        <directory>.</directory>  +          <excludes>  +            <exclude>target/**</exclude>  +          </excludes>  +      </resource>  +    </resources>  +  </build>  + </project></font>  +Next:  + <font size=2>mvn package  + mvn install</font>  +   +I'm yet to test this but the '''libtensorflow_jni_gpu-1.15.0.jar''' at least has all the files needed.     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 81:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 117:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 133:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 143:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎Build so package

← Older revision Revision as of 17:00, 9 March 2020 (18 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>  +   +===Build so package===  + <font size=2>bazel build -c opt //tensorflow/tools/lib_package:libtensorflow_jni.tar.gz</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 54:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 90:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 106:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 116:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎Install JAR to local Maven Repository

← Older revision Revision as of 17:37, 5 March 2020 (16 intermediate revisions by the same user not shown)Line 1: Line 1:  +==<font color='blue'>Why</font>==  +Why modify TF JNI?  +* Add TF features that are still missing in TF for Java like '''feeding directly from GPU memory''' thus saving time on back and forth CPU-GPU transfers if you, say, run data through a custom CUDA kernel first.  +  ==<font color='blue'>About</font>== ==<font color='blue'>About</font>==  +  +Notes on how to build TF JNI, where to modify JNI if needed, install to local maven and setup a project that will use modified native functions.     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 37: Line 43:     * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. * '''xml''' and '''jar''' will be taken care of by '''mvn''' command. −* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"+* '''so''' will have to be in the library path. Link or copy to '''/usr/lib/''' or go with "java -Djava.library.path=...". Example:  + <font size=2># /usr/lib is in the default java.library.path  + sudo ln -sf ~/GIT/tensorflow-1.15.0/bazel-bin/tensorflow/java/libtensorflow_jni.so /usr/lib/</font>     ==<font color='blue'>Install JAR to local Maven Repository</font>== ==<font color='blue'>Install JAR to local Maven Repository</font>== Line 43: Line 51:     How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +  +<font color=green size='4'>After '''*_jni.so''' is linked and '''jar''' installed one can resume normal development. See below what to add to your project's pom.xml</font>     ==<font color='blue'>Modify TF JNI functions</font>== ==<font color='blue'>Modify TF JNI functions</font>== Line 77: Line 87:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Line 92: Line 103:  ===libtensorflow=== ===libtensorflow===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow</artifactId>    <artifactId>libtensorflow</artifactId>     <version>1.15.0</version>    <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains Java classes. Archive contains Java classes. Line 102: Line 113:  ===libtensorflow_jni_gpu=== ===libtensorflow_jni_gpu===  Record in pom.xml: Record in pom.xml: −  <dependency>+  <font size=2><dependency>     <groupId>org.tensorflow</groupId>    <groupId>org.tensorflow</groupId>     <artifactId>libtensorflow_jni_gpu</artifactId>    <artifactId>libtensorflow_jni_gpu</artifactId> −   <version>1.14.0</version>+   <version>1.15.0</version> −  </dependency>+  </dependency></font>     Archive contains native library: Archive contains native library: Oleg

‎Java Maven project in Eclipse

← Older revision Revision as of 17:12, 5 March 2020 Line 77: Line 77:     ...    ...    </project></font>   </project></font>  +* Write code as usual     ===Basic example code=== ===Basic example code=== Oleg

← Older revision Revision as of 17:11, 5 March 2020 Line 3: Line 3:  What software components are still missing that you cannot live without? What software components are still missing that you cannot live without?     +{{Legacy}}     ==Software== ==Software== Line 18: Line 19:  * [[http://opensoundcontrol.org OpenSound Control]] [[http://en.wikipedia.org/wiki/OpenSound_Control  OpenSound Control on wikipedia]], [[http://www.linuxjournal.com/content/introduction-osc introduction to OSC on linuxjournal]] * [[http://opensoundcontrol.org OpenSound Control]] [[http://en.wikipedia.org/wiki/OpenSound_Control  OpenSound Control on wikipedia]], [[http://www.linuxjournal.com/content/introduction-osc introduction to OSC on linuxjournal]]  * CF Ethernet card support [[http://www.pocketpcfaq.com/peripherals/cfethernet.htm CF 10/100 Ethernet cards list]] * CF Ethernet card support [[http://www.pocketpcfaq.com/peripherals/cfethernet.htm CF 10/100 Ethernet cards list]] −* CF WiFi card support [[http://www.google.ru/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fwww.linksys.com%2Fservlet%2FSatellite%3Fchildpagename%3DUS%252FLayout%26packedargs%3Dc%253DL_Product_C2%2526cid%253D1115416826419%26pagename%3DLinksys%252FCommon%252FVisitorWrapper&ei=2CkjSb2pB4PG0gXtx92CAg&usg=AFQjCNFGaaRKL7uwQ7UNQQsxbdf9mrsoeA&sig2=wYBsCu8pfOyTJt57UW57EA WCF54G]]   * CF USB card support [[http://www.twin-paradox.com/SEPDA.html SolarExpress]] [[http://www.ratocsystems.com/english/products/CFU2U.html CFU2]] * CF USB card support [[http://www.twin-paradox.com/SEPDA.html SolarExpress]] [[http://www.ratocsystems.com/english/products/CFU2U.html CFU2]]  * External USB Display support [http://gizmodo.com/5116061/mimo-um+750-7+inch-usb-display-lightning-review Mimo UM-750 7-inch]. * External USB Display support [http://gizmodo.com/5116061/mimo-um+750-7+inch-usb-display-lightning-review Mimo UM-750 7-inch]. Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (16 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>==     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 14: Line 14:  There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.    −==Install==+==<font color='blue'>Install</font>==  In Kubuntu: In Kubuntu:  * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] Line 21: Line 21:  Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]]. Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].    −==Build==+==<font color='blue'>Build</font>==    <font size=2>cd ~/git/tensorflow-1.15.0   <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA Line 28: Line 28:  With TF, bazel tends to rebuild everything from scratch - takes a ton of time. With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  Is it because it gets restarted after idle timeout or something else? A somewhat solution might be Is it because it gets restarted after idle timeout or something else? A somewhat solution might be −  <font size=1>at launch bazel starts its server which, to prevent it, add to ~/.bazelrc:+  <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:    startup --max_idle_secs=0</font>   startup --max_idle_secs=0</font>    −Artifacts of interest in bazel-bin/tensorflow/java/:+Artifacts of interest are in '''bazel-bin/tensorflow/java/''': −  libtensorflow_jni.so+  <font size=2>'''libtensorflow_jni.so''' −  libtensorflow.jar+  '''libtensorflow.jar''' −  pom.xml+  '''pom.xml'''</font>    −* '''xml''' and '''jar''' are taken care of.+* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Install TF to local maven repo==+==<font color='blue'>Install JAR to local Maven Repository</font>== −  mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml+  <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>    −How to uninstall maven local repo - and switch back to official versions form Central:+How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' − https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install     −==Modify TF JNI==+==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc    −* rebuild and reinstall to mvn+Rebuild and Reinstall.  +   +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>    −==Java maven project in Eclipse==+==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  * Create a new maven project * Create a new maven project  * Edit pom.xml: * Edit pom.xml: −  <project>+  <font size=2><project>     ...    ...     <dependencies>    <dependencies> Line 72: Line 76:     </dependencies>    </dependencies>     ...    ... −  </project>+  </project></font>     ===Basic example code=== ===Basic example code===  tfhello.java: tfhello.java: −  import org.tensorflow.TensorFlow;+  <font size=2>import org.tensorflow.TensorFlow;             public class tfhello{   public class tfhello{ Line 82: Line 86:    System.out.println(TensorFlow.version());   System.out.println(TensorFlow.version());    }   } −  }+  }</font>  +   +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +   +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +   +Archive contains Java classes.  +   +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +   +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (17 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>==     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 14: Line 14:  There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.    −==Install==+==<font color='blue'>Install</font>==  In Kubuntu: In Kubuntu:  * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] Line 21: Line 21:  Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]]. Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].    −==Build==+==<font color='blue'>Build</font>== −  cd ~/git/tensorflow-1.15.0+  <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA −  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom+  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>    −With tensorflow bazel (when restarted?) tends to rebuild everything from scratch which takes a ton of time. Bazel starts its server which, to prevent it, add to ~/.bazelrc:+With TF, bazel tends to rebuild everything from scratch - takes a ton of time. −  startup --max_idle_secs=0+Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  +  startup --max_idle_secs=0</font>    −Artifacts of interest in bazel-bin/tensorflow/java/:+Artifacts of interest are in '''bazel-bin/tensorflow/java/''': −  libtensorflow_jni.so+  <font size=2>'''libtensorflow_jni.so''' −  libtensorflow.jar+  '''libtensorflow.jar''' −  pom.xml+  '''pom.xml'''</font>    −* '''xml''' and '''jar''' are taken care of.+* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Install TF to local maven repo==+==<font color='blue'>Install JAR to local Maven Repository</font>== −  mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml+  <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>    −How to uninstall maven local repo - and switch back to official versions form Central:+How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' − https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install     −==Modify TF JNI==+==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc    −* rebuild and reinstall to mvn+Rebuild and Reinstall.  +   +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>    −==Java maven project in Eclipse==+==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  * Create a new maven project * Create a new maven project  * Edit pom.xml: * Edit pom.xml: −  <project>+  <font size=2><project>     ...    ...     <dependencies>    <dependencies> Line 70: Line 76:     </dependencies>    </dependencies>     ...    ... −  </project>+  </project></font>     ===Basic example code=== ===Basic example code===  tfhello.java: tfhello.java: −  import org.tensorflow.TensorFlow;+  <font size=2>import org.tensorflow.TensorFlow;             public class tfhello{   public class tfhello{ Line 80: Line 86:    System.out.println(TensorFlow.version());   System.out.println(TensorFlow.version());    }   } −  }+  }</font>  +   +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +   +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +   +Archive contains Java classes.  +   +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +   +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (18 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>==     Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. Line 14: Line 14:  There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless.    −==Install==+==<font color='blue'>Install</font>==  In Kubuntu: In Kubuntu:  * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] * Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0]  * Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2] * Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]    −==Build==+Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]]. −  cd ~/git/tensorflow-1.15.0+   +==<font color='blue'>Build</font>==  +  <font size=2>cd ~/git/tensorflow-1.15.0    ./configure # do not forget CUDA   ./configure # do not forget CUDA −  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom+  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>    −With tensorflow bazel (when restarted?) tends to rebuild everything from scratch which takes a ton of time. Bazel starts its server which, to prevent it, add to ~/.bazelrc:+With TF, bazel tends to rebuild everything from scratch - takes a ton of time. −  startup --max_idle_secs=0+Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  +  startup --max_idle_secs=0</font>    −Artifacts of interest in bazel-bin/tensorflow/java/:+Artifacts of interest are in '''bazel-bin/tensorflow/java/''': −  libtensorflow_jni.so+  <font size=2>'''libtensorflow_jni.so''' −  libtensorflow.jar+  '''libtensorflow.jar''' −  pom.xml+  '''pom.xml'''</font>    −* '''xml''' and '''jar''' are taken care of.+* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Install TF to local maven repo==+==<font color='blue'>Install JAR to local Maven Repository</font>== −  mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml+  <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>    −How to uninstall maven local repo - and switch back to official versions form Central:+How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow''' − https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install     −==Modify TF JNI==+==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc    −* rebuild and reinstall to mvn+Rebuild and Reinstall.  +   +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>    −==Java maven project in Eclipse==+==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  * Create a new maven project * Create a new maven project  * Edit pom.xml: * Edit pom.xml: −  <project>+  <font size=2><project>     ...    ...     <dependencies>    <dependencies> Line 68: Line 76:     </dependencies>    </dependencies>     ...    ... −  </project>+  </project></font>     ===Basic example code=== ===Basic example code===  tfhello.java: tfhello.java: −  import org.tensorflow.TensorFlow;+  <font size=2>import org.tensorflow.TensorFlow;             public class tfhello{   public class tfhello{ Line 78: Line 86:    System.out.println(TensorFlow.version());   System.out.println(TensorFlow.version());    }   } −  }+  }</font>  +   +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +   +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +   +Archive contains Java classes.  +   +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +   +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (27 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −And this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md].+Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article.  +Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md].    −How to: +These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''.   −* Install things  −* Build libtensorflow.jar, libtensorflow_jni.so  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2  −* Modify Tensorflow JNI functions  −* Create a maven project and use modified JNI     −==Install==+How to: −* In Kubuntu+* '''Build TF JNI''' - libtensorflow.jar, libtensorflow_jni.so and pom.xml −* Get TF 1.15.0 - git or archive+* '''Add TF JAR to local Maven''' which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository] −* Install bazel 0.25.2 - see *.deb in releases+* '''Modify TF JNI functions'''  +* '''Create Elipse project'''    −==Build==+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. − cd ~/git/tensorflow-1.15.0  − ./configure  − bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni  − bazel build -c opt //tensorflow/java:pom  − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml     −Artifacts of interest in bazel-bin/tensorflow/java/:+==<font color='blue'>Install</font>== − libtensorflow_jni.so+In Kubuntu: − libtensorflow.jar+* Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0] − pom.xml+* Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]    −* '''xml''' and '''jar''' are taken care of.+Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].  +   +==<font color='blue'>Build</font>==  + <font size=2>cd ~/git/tensorflow-1.15.0  + ./configure # do not forget CUDA  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>  +   +With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  +Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0</font>  +   +Artifacts of interest are in '''bazel-bin/tensorflow/java/''':  + <font size=2>'''libtensorflow_jni.so'''  + '''libtensorflow.jar'''  + '''pom.xml'''</font>  +   +* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Modify TF JNI==+==<font color='blue'>Install JAR to local Maven Repository</font>==  + <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>  +   +How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +   +==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc    −* rebuild and reinstall to mvn+Rebuild and Reinstall.    −==Java maven project in Eclipse==+The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>  +   +==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  * Create a new maven project * Create a new maven project  * Edit pom.xml: * Edit pom.xml: −  <project>+  <font size=2><project>     ...    ...     <dependencies>    <dependencies> Line 58: Line 76:     </dependencies>    </dependencies>     ...    ... −  </project>+  </project></font>     ===Basic example code=== ===Basic example code===  tfhello.java: tfhello.java: −  import org.tensorflow.TensorFlow;+  <font size=2>import org.tensorflow.TensorFlow;             public class tfhello{   public class tfhello{ Line 68: Line 86:    System.out.println(TensorFlow.version());   System.out.println(TensorFlow.version());    }   } −  }+  }</font>  +   +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +   +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +   +Archive contains Java classes.  +   +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +   +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎A few words on TF in Maven Central repository

Show changes Oleg