‎A few words on TF in Maven Central repository

Show changes Oleg

‎A few words on TF in Maven Central repository

← Older revision Revision as of 17:02, 5 March 2020 (37 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2  −* Modify Tensorflow JNI functions  −* Create a maven project and use modified JNI     −==Install==+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''.   −* In Kubuntu  −* Get TF 1.15.0 - git or archive  −* Install bazel 0.25.2 - see *.deb in releases     −==Build==+How to: − cd ~/git/tensorflow-1.15.0+* '''Build TF JNI''' - libtensorflow.jar, libtensorflow_jni.so and pom.xml − ./configure+* '''Add TF JAR to local Maven''' which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository] − bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+* '''Modify TF JNI functions''' − bazel build -c opt //tensorflow/java:pom+* '''Create Elipse project''' − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml     −Artifacts of interest in bazel-bin/tensorflow/java/:+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. − libtensorflow_jni.so  − libtensorflow.jar  − pom.xml     −* '''xml''' and '''jar''' are taken care of.+==<font color='blue'>Install</font>==  +In Kubuntu:  +* Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0]  +* Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]  +   +Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].  +   +==<font color='blue'>Build</font>==  + <font size=2>cd ~/git/tensorflow-1.15.0  + ./configure # do not forget CUDA  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>  +   +With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  +Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0</font>  +   +Artifacts of interest are in '''bazel-bin/tensorflow/java/''':  + <font size=2>'''libtensorflow_jni.so'''  + '''libtensorflow.jar'''  + '''pom.xml'''</font>  +   +* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Modify TF JNI==+==<font color='blue'>Install JAR to local Maven Repository</font>==  + <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>  +   +How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +   +==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +Rebuild and Reinstall.  +  +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>  +  +==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  +* Create a new maven project  +* Edit pom.xml:  + <font size=2><project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project></font>  +  +===Basic example code===  +tfhello.java:  + <font size=2>import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + }</font>  +  +==<font color='blue'>A few words on TF in Maven Central repository</font>==  +  +===libtensorflow===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow</artifactId>  +  <version>1.15.0</version>  + </dependency>  +  +Archive contains Java classes.  +  +===libtensorflow_jni_gpu===  +Record in pom.xml:  + <dependency>  +  <groupId>org.tensorflow</groupId>  +  <artifactId>libtensorflow_jni_gpu</artifactId>  +  <version>1.14.0</version>  + </dependency>  +  +Archive contains native library:  + <font size=2>├── META-INF  + │   ├── MANIFEST.MF  + │   └── maven  + │      └── org.tensorflow  + │          └── libtensorflow_jni_gpu  + │              ├── pom.properties  + │              └── pom.xml  + └── org  +    └── tensorflow  +        └── native  +            ├── linux-x86_64  +            │   ├── '''libtensorflow_framework.so.1'''  +            │   ├── '''libtensorflow_jni.so'''  +            │   ├── LICENSE  +            │   └── THIRD_PARTY_TF_JNI_LICENSES  +            └── windows-x86_64  +                ├── LICENSE  +                └── tensorflow_jni.dll</font> Oleg

‎Java Maven project in Eclipse

← Older revision Revision as of 01:28, 5 March 2020 (35 intermediate revisions by the same user not shown)Line 1: Line 1: −==About==+==<font color='blue'>About</font>== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2  −* Modify Tensorflow JNI functions  −* Create a maven project and use modified JNI     −==Install==+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''.   −* In Kubuntu  −* Get TF 1.15.0 - git or archive  −* Install bazel 0.25.2 - see *.deb in releases     −==Build==+How to: − cd ~/git/tensorflow-1.15.0+* '''Build TF JNI''' - libtensorflow.jar, libtensorflow_jni.so and pom.xml − ./configure+* '''Add TF JAR to local Maven''' which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository] − bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+* '''Modify TF JNI functions''' − bazel build -c opt //tensorflow/java:pom+* '''Create Elipse project''' − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml     −Artifacts of interest in bazel-bin/tensorflow/java/:+There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. Seems useless. − libtensorflow_jni.so  − libtensorflow.jar  − pom.xml     −* '''xml''' and '''jar''' are taken care of.+==<font color='blue'>Install</font>==  +In Kubuntu:  +* Get [https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz TensorFlow-1.15.0]  +* Install [https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel_0.25.2-linux-x86_64.deb bazel 0.25.2]  +   +Based on [[Feeding_Tensorflow_from_GPU|Feeding Tensorflow from GPU]].  +   +==<font color='blue'>Build</font>==  + <font size=2>cd ~/git/tensorflow-1.15.0  + ./configure # do not forget CUDA  + bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom</font>  +   +With TF, bazel tends to rebuild everything from scratch - takes a ton of time.  +Is it because it gets restarted after idle timeout or something else? A somewhat solution might be  + <font size=2>At launch bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0</font>  +   +Artifacts of interest are in '''bazel-bin/tensorflow/java/''':  + <font size=2>'''libtensorflow_jni.so'''  + '''libtensorflow.jar'''  + '''pom.xml'''</font>  +   +* '''xml''' and '''jar''' will be taken care of by '''mvn''' command.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"    −==Modify TF JNI==+==<font color='blue'>Install JAR to local Maven Repository</font>==  + <font size=2>~/GIT/tensorflow-1.15.0$ <b>mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml</b></font>  +   +How to uninstall maven local repo - and switch back to official versions from Maven Central - [https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install this link]. Or remove unneeded stuff from '''~/.m2/repository/org/tensorflow'''  +   +==<font color='blue'>Modify TF JNI functions</font>==  For example, one wants to create a new function in org.tensorflow.TensorFlow package. For example, one wants to create a new function in org.tensorflow.TensorFlow package. −Then, see:+Then see inside: −  '''tensorflow/java/src/main/java/org/tensorflow/'''+  <font size=2>'''tensorflow/java/src/main/java/org/tensorflow/''' −  '''tensorflow/java/src/main/native/'''+  '''tensorflow/java/src/main/native/'''</font> − +Three places:  * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java * add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +Rebuild and Reinstall.  +  +The native header files seem to be regenerated but I haven't tested if they are actually used (need to test).  +In function naming - avoid underscores, e.g.:  + <font size=2>Java_org_tensorflow_TensorFlow_&lt;Name&gt;</font>  +  +==<font color='blue'>Java Maven project in Eclipse</font>==  +Nothing special.  +* Create a new maven project  +* Edit pom.xml:  + <font size=2><project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project></font>  +  +===Basic example code===  +tfhello.java:  + <font size=2>import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + }</font> Oleg

‎About

← Older revision Revision as of 00:26, 5 March 2020 (13 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About== −Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].     −How to:  +Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 Build TensorFlow 2.0 for Java on Windows] article. −* Install things+Also this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md]. −* Build libtensorflow.jar, libtensorflow_jni.so+  −* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2+These instructions are for '''Linux''' and old '''TensorFlow 1.15.0'''. −* Modify Tensorflow JNI functions+  −* Create a maven project and use modified JNI+How to:  +* Build TF JNI - '''libtensorflow.jar, libtensorflow_jni.so'''  +* Add '''libtensorflow.jar''' to local Maven repository which will override the [https://mvnrepository.com/artifact/org.tensorflow/tensorflow Central Maven Repository]  +* Modify TF JNI functions  +* Create a Maven project and use modified TF JNI  +   +There's [https://github.com/bytedeco/javacpp-presets JavaCPP Presets] project. But they fail to clearly explain what it does.     ==Install== ==Install== Line 16: Line 21:  ==Build== ==Build==    cd ~/git/tensorflow-1.15.0   cd ~/git/tensorflow-1.15.0 −  ./configure+  ./configure # do not forget CUDA −  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni+  bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni //tensorflow/java:pom − bazel build -c opt //tensorflow/java:pom+  − mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml+With tensorflow bazel (when restarted?) tends to rebuild everything from scratch which takes a ton of time. Bazel starts its server which, to prevent it, add to ~/.bazelrc:  + startup --max_idle_secs=0     Artifacts of interest in bazel-bin/tensorflow/java/: Artifacts of interest in bazel-bin/tensorflow/java/: Line 28: Line 34:  * '''xml''' and '''jar''' are taken care of. * '''xml''' and '''jar''' are taken care of.  * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path" * '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"  +  +==Install TF to local maven repo==  + mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml  +  +How to uninstall maven local repo - and switch back to official versions form Central:  + https://stackoverflow.com/questions/15358851/how-to-remove-jar-file-from-local-maven-repository-which-was-added-with-install     ==Modify TF JNI== ==Modify TF JNI== Line 38: Line 50:  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +* rebuild and reinstall to mvn  +  +==Java maven project in Eclipse==  +* Create a new maven project  +* Edit pom.xml:  + <project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project>  +  +===Basic example code===  +tfhello.java:  + import org.tensorflow.TensorFlow;  +  + public class tfhello{  + public static void main(String[] args){  + System.out.println(TensorFlow.version());  + }  + } Oleg

‎Java maven project in Eclipse

← Older revision Revision as of 23:23, 4 March 2020 (6 intermediate revisions by the same user not shown)Line 1: Line 1:  ==About== ==About==  Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this]. Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].  +  +And this one - [https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md tensorflow/java/README.md].     How to:   How to:   Line 38: Line 40:  * add to header file tensorflow/java/src/main/native/tensorflow_jni.h * add to header file tensorflow/java/src/main/native/tensorflow_jni.h  * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc * add to c file tensorflow/java/src/main/native/tensorflow_jni.cc  +  +* rebuild and reinstall to mvn  +  +==Java maven project in Eclipse==  +* Create a new maven project  +* Edit pom.xml:  + <project>  +  ...  +  <dependencies>  +    ...  +    <b><dependency>  +      <groupId>org.tensorflow</groupId>  +      <artifactId>libtensorflow</artifactId>  +      <version>1.15.0</version>  +    </dependency></b>  +    ...  +  </dependencies>  +  ...  + </project> Oleg

Created page with "==About== Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this]. How to: * Install things * Build libtensorflow.jar, li..."

New page

==About==
Based on [https://medium.com/@maxime.durand.54/build-tennsorflow-2-0-for-java-on-windows-2ab51b9cac45 this].

How to:
* Install things
* Build libtensorflow.jar, libtensorflow_jni.so
* Install pom.xml and libtensorflow.jar to local maven repo in ~/.m2
* Modify Tensorflow JNI functions
* Create a maven project and use modified JNI

==Install==
* In Kubuntu
* Get TF 1.15.0 - git or archive
* Install bazel 0.25.2 - see *.deb in releases

==Build==
cd ~/git/tensorflow-1.15.0
./configure
bazel build -c opt //tensorflow/java:tensorflow //tensorflow/java:libtensorflow_jni
bazel build -c opt //tensorflow/java:pom
mvn install:install-file -Dfile=bazel-bin/tensorflow/java/libtensorflow.jar -DpomFile=bazel-bin/tensorflow/java/pom.xml

Artifacts of interest in bazel-bin/tensorflow/java/:
libtensorflow_jni.so
libtensorflow.jar
pom.xml

* '''xml''' and '''jar''' are taken care of.
* '''so''' will have to be in the library path - set LD_LIBRARY_PATH or PATH or go "java -Djava.library.path"

==Modify TF JNI==
For example, one wants to create a new function in org.tensorflow.TensorFlow package.
Then, see:
'''tensorflow/java/src/main/java/org/tensorflow/'''
'''tensorflow/java/src/main/native/'''

* add native method to tensorflow/java/src/main/java/org/tensorflow/TensorFlow.java
* add to header file tensorflow/java/src/main/native/tensorflow_jni.h
* add to c file tensorflow/java/src/main/native/tensorflow_jni.cc Oleg

AndreyFilippov committed changes to the Elphel git project :
updated tested GPU image conversion with selectable kernel source

AndreyFilippov committed changes to the Elphel git project :
updated pom to jcuda 10.1.0, moved resources to match src tree

Oleg Dzhimiev committed changes to the Elphel git project :
jcuda testing

AndreyFilippov committed changes to the Elphel git project :
improving/fixing bugs in the sensors calibration.

AndreyFilippov committed changes to the Elphel git project :
improving residual sensor correction

AndreyFilippov committed changes to the Elphel git project :
Merge branch 'lwir-distort' of git.elphel.com:Elphel/imagej-elphel into lwir-distort

AndreyFilippov committed changes to the Elphel git project :
working on dual-modal calibration

Andrey Filippov committed changes to the Elphel git project :
corrected wait time for short movements

Andrey Filippov committed changes to the Elphel git project :
limiting number of frames

Andrey Filippov committed changes to the Elphel git project :
limiting number of consecutive frames

‎For Java

← Older revision Revision as of 18:48, 20 February 2020 (One intermediate revision by the same user not shown)Line 113: Line 113:    cd ..   cd ..    ./run_tf_elphel.sh 1</font>   ./run_tf_elphel.sh 1</font>  +  +==For Java==  +* https://www.tensorflow.org/install/lang_java which leads to a more detailed:  +* https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/README.md  +  +In short, to Build TF for Java from source:  + <font size=2>./configure  + bazel build --config opt \  +  //tensorflow/java:tensorflow \  +  //tensorflow/java:libtensorflow_jni</font> Oleg

AndreyFilippov committed changes to the Elphel git project :
restored LWIR_ACQUIRE save directory

AndreyFilippov committed changes to the Elphel git project :
reordered camera url

AndreyFilippov committed changes to the Elphel git project :
implementing initial orientation estimation, full window for eo acquisition