I'm delighted to share with you the process you would take to setup tensorflow on your machine for open source contribution.
This usually takes days to setup and you don't really get a hand held article on the internet for this.
Soooooo! i've gone out of my way to make this step easy for the beginner who wants to get their hands dirty. Like i'm so exhausted right now 🥴, it's going to be a long one so just grab your coke and popcorn.
In this case i would be using ubuntu 22.04 but the process would be similar on any machine since we would be making most changes in docker.
🥷 Fork & clone tensorflow repo.
⚙️ Setup Development environment for contribution.
🫸 Push and create pull request.
🥷 Fork & clone tensorflow repo
Head to the tensorflow github page and fork the repo.
After doing this you would have your version of the repo for contribution purpose
⚙️ Setup Development environment for contribution
Next, would be to download the repo in zip format or clone the repo in your terminal (Http or SSH)
$ git clone https://github.com/MrChike/tensorflow.git
Cloning into 'tensorflow'...
After cloning, we would want to connect to the parent repo so to be updated with the latest changes.
$ git remote add upstream git@github.com:tensorflow/tensorflow.git
$ git remote -v
origin git@github.com:MrChike/tensorflow.git (fetch)
origin git@github.com:MrChike/tensorflow.git (push)
upstream git@github.com:tensorflow/tensorflow.git (fetch)
upstream git@github.com:tensorflow/tensorflow.git (push)
$ git fetch upstream
By doing this we are now connected and up-to-date with tensorflow repo and can checkout the whatever release we would want to contribute to
$ git branch -r
origin/HEAD -> origin/master
origin/master
upstream/LakshmiKalaKadali-patch-1
upstream/Lakshmikala_systemconfig
upstream/Surya_-MatrixSetDiagV3_checkfail
upstream/SuryanarayanaY-patch-2
:
$ git tag
v2.17.1
v2.18.0
v2.18.0-rc0
$ git checkout v2.17.1
Updating files: 100% (11816/11816), done.
Note: switching to 'v2.17.1'.
Let's take it a step further by introducing docker into the picture, Why? 🐔Because! it's important we isolate the environment and it's build dependencies away from our host machine which gives us a playground to break things and experiment for the fun of it. So we would start by mounting a volume from our host machine to the docker container.
$ docker pull tensorflow/tensorflow:latest
$ docker run --name tensorflow_contribution -it -v $(pwd)/:/tensorflow tensorflow/tensorflow:latest /bin/bash
$ apt update && apt upgrade
$ python --version
root@f56d002b5afc:/# python --version
Python 3.11.0rc1
$ apt install python3-dev python3-pip
$ pip install -U pip
$ pip --version
root@f56d002b5afc:/# pip --version
pip 24.3.1 from /usr/local/lib/python3.11/dist-packages/pip (python 3.11)
# if pip doesn't match your python version which in this case is v3.11 then you would have to update pip to match it.
$ python3.11 -m pip install pip==24.3.1
________ _______________
___ __/__________________________________ ____/__ /________ __
__ / _ _ \_ __ \_ ___/ __ \_ ___/_ /_ __ /_ __ \_ | /| / /
_ / / __/ / / /(__ )/ /_/ / / _ __/ _ / / /_/ /_ |/ |/ /
/_/ \___//_/ /_//____/ \____//_/ /_/ /_/ \____/____/|__/
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.
To avoid this, run the container by specifying your user's userid:
$ docker run -u $(id -u):$(id -g) args...
root@2e306f66dcfe:/#
You could crank it up a notch with vscode giving your IDE access to your docker container with the Attach Visual Studio Code option or the equivalent in your favourite IDE and make some changes
Hope you not getting b🙄red!.Now that we have our cloned repo s ynced with the docker container. We could make changes in the tensorflow/ folder and the changes would reflect on the host machine.
Well now we got it all setup, you would think we all good right? Psyche!!!!! 😂. Stay with me, coz we still gotta loooong way to go amigo. So lest I deviate, back to business...
The next step would be to run the configure file to build the changes we've made with ./configure
$ cd tensorflow/
$ ./configure
root@2e306f66dcfe:/tensorflow# ./configure
Cannot find bazel. Please install bazel/bazelisk.
In the dev world, a new error is a sign of progress...
So what's bazel? It's a build tool developed by Google. It helps automate the process of compiling and testing code. So we would be installing it as instructed in the documentation.
$ apt install apt-transport-https curl gnupg -y
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg
mv bazel-archive-keyring.gpg /usr/share/keyrings
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
$ apt update && apt install bazel
Now we try to run the configure file again
root@f56d002b5afc:/tensorflow# ./configure
WARNING: current bazel installation is not a release version.
Please specify the location of python. [Default is /usr/bin/python3]:
Found possible Python library paths:
/usr/lib/python3/dist-packages
/usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use. Default is [/usr/lib/python3/dist-packages]
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: N
No CUDA support will be enabled for TensorFlow.
Do you want to use Clang to build TensorFlow? [Y/n]: Y
Clang will be used to compile TensorFlow.
Please specify the path to clang executable. [Default is ]:
Invalid clang path. cannot be found. Note that TensorFlow now requires clang to compile. You may override this behavior by setting TF_NEED_CLANG=0
Please specify the path to clang executable. [Default is ]:
At this point we've meet another blocker, so what's clang? Clang is a compiler for the C, C++, and Objective-C programming languages used as the compiler for building TensorFlow from source which helps compile the C++ components of TensorFlow efficiently and provides useful diagnostics. So we would be installing it from the guide
$ wget https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.2/clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04.tar.xz
$ tar -xvf clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04.tar.xz
$ cp -r clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04/* /usr
$ clang --version
root@f56d002b5afc:/tensorflow# clang --version
clang version 17.0.2 (https://github.com/llvm/llvm-project b2417f51dbbd7435eb3aaf203de24de6754da50e)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ which clang
root@f56d002b5afc:/tensorflow# which clang
/usr/bin/clang
Now that we know path in which clang has been installed, let's run the configure file again and include the path
$ ./configure
root@f56d002b5afc:/tensorflow# ./configure
WARNING: current bazel installation is not a release version.
Please specify the location of python. [Default is /usr/bin/python3]:
Found possible Python library paths:
/usr/lib/python3/dist-packages
/usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use. Default is [/usr/lib/python3/dist-packages]
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: N
No CUDA support will be enabled for TensorFlow.
Do you want to use Clang to build TensorFlow? [Y/n]: Y
Clang will be used to compile TensorFlow.
Please specify the path to clang executable. [Default is /usr/bin/clang]:
You have Clang 17.0.2 installed.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
--config=mkl # Build with MKL support.
--config=mkl_aarch64 # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
--config=monolithic # Config for mostly static monolithic build.
--config=numa # Build with NUMA support.
--config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
--config=v1 # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
--config=nogcp # Disable GCP support.
--config=nonccl # Disable NVIDIA NCCL support.
Configuration finished
Now that we have clang installed let's build our code to reflect changes. This is the format bazel usually takes
bazel build //path/to/folder:file_target --repo_env=WHEEL_NAME=your_personalized_build_name.
To add some knowlege to your database, When you run pip install tensorflow, you typically don’t see a .whl extension in the command or output because pip handles the download, extraction, and installation of the wheel file seamlessly in the background. However, behind the scenes, pip is indeed downloading a .whl file from PyPI and using it for the installation. So to continue with our bazel installation
$ apt install patchelf
$ bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow_cpu
root@f56d002b5afc:/tensorflow# bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow_cpu
ERROR: The project you're trying to build requires Bazel 6.5.0 (specified in /tensorflow/.bazelversion), but it wasn't found in /usr/bin.
You can install the required Bazel version via apt:
sudo apt update && sudo apt install bazel-6.5.0
If this doesn't work, check Bazel's installation instructions for help:
https://bazel.build/install/ubuntu
We need to install the exact bazel version the tensorflow project is using as suggested in the terminal and rerun the bazel command
$ apt update && apt install bazel-6.5.0
$ bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow_cpu
root@f56d002b5afc:/tensorflow# bazel build //tensorflow/tools/pip_package:wheel --repo_env=WHEEL_NAME=tensorflow_cpu
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Analyzing: target //tensorflow/tools/pip_package:wheel (544 packages loaded, 22399\
targets configured)
INFO: Found 1 target...
[809 / 6,002] 2 actions running
Compiling src/idl_gen_php.cpp [for tool]; 3s local
Compiling src/ssl/tls_record.cc [for tool]; 1s local
INFO: Elapsed time: 368.128s, Critical Path: 317.42s
INFO: 5 processes: 1 internal, 4 local.
INFO: Build completed successfully, 5 total actions
NB: There are three versions for possible build.. CPU, GPU & TPU. This tutorial focused on CPU and for the first build it took me 24hrs+, and upon subsequent builds in minutes because the first build has been cached in the bazel folder.
So now your changes are being compiled and after that we can install the compiled version in our tensorflow changes in the terminal.
pip install path/to/wheel
$ pip uninstall tensorflow # If you are removing an already installed custom tensorflow build, it would be that build name instead. In my case, tensorflow_cpu.
$ pip install bazel-bin/tensorflow/tools/pip_package/wheel_house/tensorflow_cpu-2.17.1-cp311-cp311-linux_x86_64.whl
$ pwd
root@f56d002b5afc:/# pwd
/
root@f56d002b5afc:/# python
Python 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-11-04 02:41:20.894095: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-11-04 02:41:21.211472: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
HELLO WORLD!
As you can see the HELLO WORLD! changes made in the tensorflow/python/keras/models.py module now reflects in our compiled version of tensorflow. The next step would be to commit our changes, push and create a pull request.
NB: The community has it's code of conduct you have to follow to push so i would suggest you familiarize yourself with it before creating your pull request. So let's assume are familiar with this and create our pull request
🫸 Push and create pull request
So now that everything is set, let's do the needful
$ git add . && git commit -m "Initial Commit" && git push
Now head to the branch you've created on your github repo and create pull request as seen in the screenshots below
Now you have completed the full tutorial.
What's left is to collaborate, contribute, solve issues and network in the community..... If you found this article helpful, i would appreciate that you engage in the comment section. 😇
Resources:
- https://emojifinder.com/
- https://www.emojisearch.app/
- https://emojipedia.org/
- https://dillinger.io/
- https://github.com/tensorflow/tensorflow
- https://annotely.com/editor
- https://www.tensorflow.org/install/source
- https://hub.docker.com/r/tensorflow/tensorflow/
- https://bazel.build/install/ubuntu
- https://www.tensorflow.org/community/contribute
- https://ezgif.com/
Top comments (0)