I have included following topics in this article,
- How to test run Mediapipe mobile app using their github code
- Details about codebase structure
- In brief about some key components of code
Compile and try out already implemented example projects on android
Build docker image using command
$ docker build . -t mediapipe
Run docker container over image
$ docker run -i -t IMAGE_ID /bin/bash
Inside the container set up Android using command
$ bash ./setup_android_sdk_and_ndk.sh ~/Android/Sdk ~/Android/Ndk r21
In the next step, build android binary using bazel.
$ bazel build -c opt mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu:facedetectioncpu
this command is for building facedetectioncpu project. Replace project_name (facedetectioncpu here) with the name of the project you want to build.
Name of the folders under pathmediapipe/mediapipe/examples/android/src/java/com/google/mediapipe/apps/
is name of the in built projects.
This command would take long for the first build. For any subsequent builds, it would use the cached data and shouldn't take long.apk will be stored at
/mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps
. Copy the apk file using
/project_name
sudo docker cp CONTAINER_ID:mediapipe/mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps
/project_name /home/username/downloads/.
copy this apk file to your android device and install.
or directly install from terminal using commands
$ adb install project.apk
keep the phone connected and logs could be viewed using
$ adb logcat
You can greb specific logs for example errors using
$ adb logcat | greb 'error'
Structure of Mediapipe project repo
Image at the top has general structure of the mediapipe project.
we call bazel build on BUILD file present at
mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu
This build file refers to graph present at
/mediapipe/graphs/facemeshgpu/facemeshgpu.pbtxt
This graph at
/mediapipe/graphs/facemeshgpu/facemeshgpu.pbtxt
refers to one or more calculators present at 'mediapipe/calculators/folder_name/calculator.cc`both folders
mediapipe/calculators/folder_name/.
andmediapipe/graphs/project_name/.
has BUILD files that contains information about dependencies.
Graphs
graphs could be visualised here https://viz.mediapipe.dev/
copy paste content from any of your .pbtxt file e.g
https://github.com/google/mediapipe/blob/master/mediapipe/graphs/face_detection/face_detection_full_range_mobile_gpu.pbtxt
to editor on visualiser. It would show the graph on the left side.Each box is calculator and has input and output.
This input output relations are specified in .pbtxt graph files.
Calculators
each box in graph visualiser is calculator. (or subgraph made of calculators)
calculators has process function defined that performes calculations. e.g render text on frame or render assets like spectacles at specific position on frame. e.g https://github.com/google/mediapipe/blob/6cdc6443b6a7ed662744e2a2ce2d58d9c83e6d6f/mediapipe/calculators/util/landmark_visibility_calculator.cc#L66
other than process, calculator have getcontract and open function. Open function is called once at the start and process function is called for each input again and again. see below for example.
About open vs process function in calculators
consider we need to display frame count on the camera preview. We start with 0 (for the first frame as the camera starts) and increment count for every subsequent frame. For this, we would define variable count in open function of calculator and assign value 0 to it.
We will add increment logic count = count + 1
in process function and display it on frame.
count will be incremented for every frame because process function will be called for each input, i.e frame. Open is called once at the beginning and value 0 will be assigned to count.
Modify current project
- add new calculator in project_folder inside calculator folder
- add entry in BUILD file of the project calculator folder
- create project folder inside graph folder and create .pbtxt graph for new project.
- add entry in BUILD file of project graph folder
- modify BUILD
mediapipe/examples/android/src/java/com/google/mediapipe/apps/BUILD
- compile using command
bazel build PATH TO BUILD
Top comments (0)