Introduction
In the first part of this series, we laid the groundwork by setting up Kotlin within our React Native environment, a crucial step in bringing augmented reality (AR) to our applications. This segment is designed for developers who are ready to take the next step in their AR journey, willing to learn how to integrate ARCore into their hybrid apps.
This is part two of a multi part article series about the integration of AR services via native modules into a React Native app. You will learn how to integrate ARCore and how to display a 3D-model in this article.
- Part one: Setting up a native module via Kotlin.
- Part Three: Setting up a native module via Objective-C
- Part Four: Integrating ARKit
The full code for this part is here.
What is ARCore
ARCore is Google's platform for building augmented reality (AR) experiences. It enables phones to understand and interact with the real world. Using three key capabilities – motion tracking, environmental understanding, and light estimation – ARCore makes it possible for apps to merge virtual content with the real world in a seamless and realistic way.
Add dependencies to the project
First we need to add the necessary dependencies to our project. Go into your android/app/build.gradle and add the following implementations to your dependencies section:
// AR
implementation 'com.google.ar:core:1.40.0'
implementation("io.github.sceneview:arsceneview:0.10.0")
It should look like this:
Then we have to bump our minSdkVersion to 24 in our android/build.gradle. Afterwards sync the project.
Download a model
Now we need a 3D model to display. Download one from here for example. Make sure to select the GLB version!
Add file to Android
To add a GLB model to your Android project in Android Studio, you should create an assets folder and place your GLB file there. Here's how to do it:
- Create the assets Folder:
- Right-click on the app folder in the Android Studio project view.
- Go to New > Folder > Assets Folder.
- Click Finish in the dialog that appears to create the folder.
- Add Your GLB File:
- Copy your GLB model file.
- Paste it into the assets folder you just created.
Create an Activity XML file
Now we need to create the layout file where our model will be displayed:
Locate the res (Resources) Folder: In your project hierarchy in Android Studio, it is typically found under app > src > main.
Create the layout Folder:
- Right-click on the res folder.
- Choose New > Directory.
- Name the directory layout.
- Add Your XML File:
- Right-click on the newly created layout folder.
- Choose New > Layout resource file.
- Name the file ar_activity.xml and click 'OK'.
Android Studio will open the file in Design mode, but we will need to change into Code mode. In order to do this, click on the Code button (should be in the upper right and looks like 5 vertical lines). Copy the following code into the file:
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
>
<io.github.sceneview.ar.ArSceneView
android:id="@+id/sceneView"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:layout_editor_absoluteX="25dp"
tools:layout_editor_absoluteY="16dp" />
<ProgressBar
android:id="@+id/progressBar"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_centerInParent="true"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="@+id/sceneView"
app:layout_constraintTop_toTopOf="@+id/sceneView" />
<Button
android:id="@+id/closeButton"
android:layout_width="48dp"
android:layout_height="48dp"
android:layout_alignParentTop="true"
android:layout_alignParentEnd="true"
android:layout_marginTop="30dp"
android:layout_marginEnd="30dp"
android:text="X"
android:textColor="#fff"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintTop_toTopOf="parent"
tools:layout_editor_absoluteX="365dp" />
</androidx.constraintlayout.widget.ConstraintLayout>
If there is a warning / error about a missing package `androidx.constraintlayout.widget.ConstraintLayout´ right click on it, select Show Context Actions and select Add dependency. Android Studio will then add an import and sync the project.
The button at the end is so that we can close the screen, when we want to return to the non-ar part of our app. In order for this button to be displayed correctly, we need to
Update the ARModule file
Now we can reference the created layout file in our module. Open the file ARModule.kt and replace the showAR method with the following:
`
@ReactMethod
fun showAR(fileName: String) {
val assetPath = "file:///android_asset/$fileName"
val intent = Intent(reactContext, ModelDisplayActivity::class.java)
intent.putExtra("MODEL_PATH", assetPath)
// If starting the activity from a non-activity context, set this flag
intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
reactContext.startActivity(intent)
}
We will pass the filename of our model to our native code, which will in turn load it from the assets folder and start a custom Activity to display the model. In simple terms, an activity represents a single screen with a user interface.
Create the Activity file
Right click on the com.rn3dworldexplorer folder and select New > Kotlin Class. Name the file ModelDisplayActivity.kt. Copy the following code into it:
package com.rn3dworldexplorer
import android.os.Bundle
import android.util.Log
import android.view.View
import android.widget.Button
import android.widget.ProgressBar
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import com.google.ar.core.Config
import io.github.sceneview.ar.ArSceneView
import io.github.sceneview.ar.node.ArModelNode
import io.github.sceneview.ar.node.PlacementMode
import io.github.sceneview.math.Position
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
class ModelDisplayActivity : AppCompatActivity() {
private lateinit var progressBar: ProgressBar
private lateinit var closeButton: Button
private lateinit var sceneView: ArSceneView
private lateinit var modelNode: ArModelNode
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.ar_activity)
// Initialize views
closeButton = findViewById(R.id.closeButton)
progressBar = findViewById(R.id.progressBar)
sceneView = findViewById<ArSceneView>(R.id.sceneView).apply {
this.lightEstimationMode = Config.LightEstimationMode.DISABLED
}
sceneView.onArSessionFailed = { exception: Exception ->
Log.e("ARModule", "${exception.message}")
// If AR is not available, we add the model directly to the scene for a 3D only usage
sceneView.addChild(modelNode)
}
// Set close button listener
closeButton.setOnClickListener {
finish()
}
// Load model if path is provided
intent.getStringExtra("MODEL_PATH")?.let { modelPath ->
loadModel(modelPath)
} ?: showMessage("Model path is null")
}
private fun loadModel(glbFileLocation: String) {
Log.d("ARModule", "loadModel path: $glbFileLocation")
progressBar.visibility = View.VISIBLE
CoroutineScope(Dispatchers.Main).launch {
try {
modelNode = ArModelNode(sceneView.engine, PlacementMode.INSTANT).apply {
loadModelGlbAsync(
glbFileLocation = glbFileLocation,
scaleToUnits = 1f,
centerOrigin = Position(0.0f)
)
{
sceneView.planeRenderer.isVisible = true
val materialInstance = it.materialInstances[0]
}
}
sceneView.addChild(modelNode)
modelNode.anchor()
sceneView.planeRenderer.isVisible = false
} catch (e: Exception) {
showMessage("Error loading model: ${e.message}")
Log.e("ARModule", "Error occurred: ${e.message}")
} finally {
progressBar.visibility = View.GONE
}
}
}
private fun showMessage(message: String) {
Toast.makeText(this, message, Toast.LENGTH_LONG).show()
}
}
Here is a brief overview of the file:
Properties
- UI Elements: The class defines UI elements like a ProgressBar, a Button (closeButton), and an ArSceneView (for AR content display).
- Model Node: modelNode of type ArModelNode is used to represent the 3D model in AR.
onCreate Method
- UI Setup: In the onCreate method (called when the activity is starting), it sets up the user interface from a layout resource (R.layout.ar_activity).
- Initializing Views: Finds and initializes views like the close button, progress bar, and AR scene view.
- AR Session Failure Handling: Sets an error handling mechanism for the AR session. If AR is not available, it directly adds the model to the scene for non-AR 3D viewing.
- Close Button Listener: Implements an action for the close button - when clicked, it finishes (closes) the activity.
- Model Loading: Checks if a model path is provided via the intent; if so, it calls loadModel to load the model.
loadModel Method
- Functionality: This method loads the 3D model (GLB format) for display in the AR scene.
- Coroutine: Uses a coroutine for asynchronous execution to avoid blocking the UI thread.
- Model Node Setup: Configures the modelNode with the model path, scale, and origin. Once the model is loaded, it is anchored to a position in the AR scene.
- Visibility Settings: Manages the visibility of the plane renderer and progress bar based on the model loading status.
- Error Handling: Catches exceptions, logs errors, and shows error messages if model loading fails.
showMessage Method
Functionality: A utility method to show a toast message on the screen. It's used to provide feedback to the user (like successful model loading or error messages).
Added the necessary permissions to the Androidmanifest.xml file
Replace the contents of your AndroidManifest.xml with this one:
<manifest xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.CAMERA" /> <!-- You may need these if doing any screen recording from within the app -->
<uses-feature
android:name="android.hardware.vr.headtracking"
android:required="true"
android:version="1" />
<uses-permission android:name="android.permission.NFC" />
<uses-permission android:name="android.permission.WAKE_LOCK" /> <!-- Other camera related features -->
<uses-feature android:name="android.hardware.camera" />
<uses-feature
android:name="android.hardware.camera.autofocus"
android:required="false"
tools:replace="required" />
<uses-feature android:name="android.hardware.camera.ar" /> <!-- Specifying OpenGL version or requirements -->
<uses-feature
android:glEsVersion="0x00030000"
android:required="false"
tools:node="remove"
tools:replace="required" /> <!-- Usage of accelerometer and gyroscope -->
<uses-feature
android:name="android.hardware.sensor.accelerometer"
android:required="false"
tools:replace="required" />
<uses-feature
android:name="android.hardware.sensor.gyroscope"
android:required="false"
tools:replace="required" />
<application
android:name=".MainApplication"
android:label="@string/app_name"
android:icon="@mipmap/ic_launcher"
android:roundIcon="@mipmap/ic_launcher_round"
android:allowBackup="false"
android:theme="@style/AppTheme">
<activity
android:name=".MainActivity"
android:label="@string/app_name"
android:configChanges="keyboard|keyboardHidden|orientation|screenLayout|screenSize|smallestScreenSize|uiMode"
android:launchMode="singleTask"
android:windowSoftInputMode="adjustResize"
android:exported="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
<activity android:name=".ModelDisplayActivity" />
<meta-data
android:name="com.google.ar.core"
android:value="optional"
tools:replace="android:value" />
</application>
</manifest>
We request several device features needed for AR like the gyroscope as well as register our custom activity via this line: <activity android:name=".ModelDisplayActivity" />. Not all of them may be used in the end, so it is your responsibility to choose carefully. This list just aims to be as comprehensive as possible out of convenience.
Adjust our JavaScript React code
We are finally back in the cozy surroundings of your JavaScript code. All that is left is to adjust our ARModule.tsx and our App.tsx. For the ARModule file, adjust your showAR function to this showAR(path: string): Promise<void>; to reflect the new type signature. After that, in the App.tsx you can adjust the onPress function of the button to pass the name of your chosen model. For me it is like this:
onPress={async () => await ARModule.showAR("AR-model.glb")}.
Now just press the button and it should look similar to this:
Conclusion
As we conclude this second part of our series, we have navigated through the integration of ARCore in a React Native environment via Kotlin. It can be pretty overwhelming if you haven't delved too deep into native code integration, but hopefully, it was manageable. From here on out, you can expand on the basic functionality and create your own features.
Moving forward, we will be doing the setup for native modules using Objective C in iOS in the next entry.
Top comments (0)