Some time ago the article “How to create an application on x86 Android with OpenVINO” has been posted. However, most Android devices use ARM-based chips, so we decided to port the instruction to this platform.

OpenVINO supports DL network inference on ARM platforms via ARM plugin, so there is no technical limitations to infer networks on ARM-based Android devices.

In this article we will build OpenVINO with Java binding and ARM plugin modules for ARM Android platform and create a face detection Android application. The application reads frames from camera with OpenCV, uses a DL network to recognize faces in frames and displays frames with bounding boxes.

The instruction has been tested on Ubuntu 18.04, but it could be easily adapted to other operation systems.

Build OpenVINO for Android on ARM

The procedure below explains how to build OpenVINO with Java bindings and ARM plugin for ARM Android platform.

• Install OpenJDK 8 and define JAVA_PATH:

sudo apt-get install -y openjdk-8-jdk
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

• Create a working directory and clone OpenVINO and OpenVINO Contrib repositories there. OpenVINO Contrib repository contains Java bindings and ARM plugin modules.

mkdir ~/android_ov && cd ~/android_ov
git clone --recurse-submodules --shallow-submodules --depth 1 --branch=2021.4.1 https://github.com/openvinotoolkit/openvino.git
git clone --recurse-submodules --shallow-submodules --depth 1 --branch=2021.4 https://github.com/openvinotoolkit/openvino_contrib.git

• Download and unpack Android NDK:

wget https://dl.google.com/android/repository/android-ndk-r20-linux-x86_64.zip
unzip android-ndk-r20-linux-x86_64.zip

• Create a directory and build OpenVINO for Android:

mkdir openvino/build && cd openvino/build
cmake -DCMAKE_BUILD_TYPE=Release \
      -DTHREADING=SEQ \
      -DIE_EXTRA_MODULES=.../openvino_contrib/modules \
      -DBUILD_java_api=ON \
      -DCMAKE_TOOLCHAIN_FILE=.../android-ndk-r20/build/cmake/android.toolchain.cmake \
      -DANDROID_ABI=arm64-v8a \
      -DANDROID_PLATFORM=29 \
      -DANDROID_STL=c++_shared \
      -DENABLE_SAMPLES=OFF \
      -DENABLE_OPENCV=OFF \
      -DENABLE_CLDNN=OFF \
      -DENABLE_VPU=OFF \
      -DENABLE_GNA=OFF \
      -DENABLE_MYRIAD=OFF \
      -DENABLE_TESTS=OFF \
      -DENABLE_GAPI_TESTS=OFF \
      -DENABLE_BEH_TESTS=OFF .. && make --jobs=$(nproc --all)

• OpenVINO binaries could be stripped to reduce their size:

~/android_ov/android-ndk-r20/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/aarch64-linux-android/bin/strip ../bin/aarch64/Release/lib/*.so

Create Android application

OpenVINO libraries are ready, so let’s create Android application and leverage OpenVINO libraries to infer face detection model.

• Download and install the Android Studio on your PC.
• Start a new project, choose “Empty Activity” with Java language and SDK version 23
• Modify app/src/main/AndroidManifest.xml: add “<uses-permission>” tag and “android:screenOrientation” attribute to the “<activity>” tag:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
 package="com.example.app">
    <uses-permission android:name="android.permission.CAMERA"/>
        <application
                android:allowBackup="true"
                android:icon="@mipmap/ic_launcher"
                android:label="@string/app_name"
                android:roundIcon="@mipmap/ic_launcher_round"
                android:supportsRtl="true"
                android:theme="@style/Theme.App">
        <activity android:name=".MainActivity" android:screenOrientation="landscape">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>
    </application>
</manifest>

• Create “jniLibs/arm64-v8a” directory in the “app/src/main” and copy the following files there:

~/android_ov/openvino/bin/aarch64/Release/lib/*.so
~/android_ov/android-ndk-r20/sources/cxx-stl/llvm-libc++/libs/arm64-v8a/libc++_shared.so

• Copy “~/android_ov/openvino/bin/aarch64/Release/lib/inference_engine_java_api.jar” to the “app/libs” folder
• Download OpenCV SDK for Android and unpack it
• Import OpenCV module: select “File -> New -> ImportModule”, specify a path to the unpacked SDK (e.g. “opencv-4.5.0-android-sdk/OpenCV-android-sdk/sdk”) and set module name to “ocv”.
• Some versions of android studio do not allow OpenCV to be loaded. If this happens, then:
  • create a new module (File – New – New module)
  • name it the same (e.g. “ocv”)
  • copy or replace all files from “opencv-4.5.0-android-sdk/OpenCV-android-sdk/sdk”
  • rebuild gradle project
  • install OpenCV as a dependency of the main project: “
  • File – Project Structures – Dependencies – <Yours app (e.g. app)> – Right mouse button – Module Dependency – ocv”
• Modify app/src/main/res/layout/activity_main.xml: add “JavaCameraView” for OpenCV Camera Support:

 <?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <org.opencv.android.JavaCameraView
        android:id="@+id/CameraView"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:visibility="visible" />

</androidx.constraintlayout.widget.ConstraintLayout>

• Update build.gradle file (module: <application_name>.app) – add the following lines to the “dependencies” section:

implementation "androidx.camera:camera-camera2:1.0.0"
implementation "androidx.camera:camera-lifecycle:1.0.0"
implementation "androidx.camera:camera-view:1.0.0-alpha25"
implementation files('libs/inference_engine_java_api.jar')
implementation project(path: ':ocv')

• Modify app/src/main/java/com/example/<application_name>/MainActivity.java

import androidx.annotation.NonNull;
import android.Manifest;
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.os.Environment;
import android.util.Log;
import android.widget.Toast;

import org.opencv.android.CameraActivity;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.core.Size;
import org.opencv.imgproc.Imgproc;
import org.intel.openvino.*;

import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class MainActivity extends CameraActivity implements CvCameraViewListener2 {
    private void copyFiles() {
        String[] fileNames = {MODEL_BIN, MODEL_XML, PLUGINS_XML};
        for (String fileName: fileNames) {
            String outputFilePath = modelDir + "/" + fileName;
            File outputFile = new File(outputFilePath);
            if (!outputFile.exists()) {
                try {
                    InputStream inputStream = getApplicationContext().getAssets().open(fileName);
                    OutputStream outputStream = new FileOutputStream(outputFilePath);
                    byte[] buffer = new byte[5120];
                    int length = inputStream.read(buffer);
                    while (length > 0) {
                        outputStream.write(buffer, 0, length);
                        length = inputStream.read(buffer);
                    }
                    outputStream.flush();
                    outputStream.close();
                    inputStream.close();
                } catch (Exception e) {
                    Log.e("CopyError", "Copying model has failed.");
                    System.exit(1);
                }
            }
        }
    }

    private void processNetwork() {
        // Set up camera listener.
        mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.CameraView);
        mOpenCvCameraView.setVisibility(CameraBridgeViewBase.VISIBLE);
        mOpenCvCameraView.setCvCameraViewListener(this);
        mOpenCvCameraView.setCameraPermissionGranted();

        // Initialize model
        copyFiles();
        IECore core = new IECore(modelDir + "/" + PLUGINS_XML);
        CNNNetwork net = core.ReadNetwork(modelDir + "/" + MODEL_XML);

        Map<String, InputInfo> inputsInfo = net.getInputsInfo();
        inputName = new ArrayList<String>(inputsInfo.keySet()).get(0);
        InputInfo inputInfo = inputsInfo.get(inputName);
        inputInfo.getPreProcess().setResizeAlgorithm(ResizeAlgorithm.RESIZE_BILINEAR);
        inputInfo.setPrecision(Precision.U8);

        ExecutableNetwork executableNetwork = core.LoadNetwork(net, DEVICE_NAME);
        inferRequest = executableNetwork.CreateInferRequest();

        Map<String, Data> outputsInfo = net.getOutputsInfo();
        outputName = new ArrayList<>(outputsInfo.keySet()).get(0);
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        try{
            System.loadLibrary(OPENCV_LIBRARY_NAME);
            System.loadLibrary(IECore.NATIVE_LIBRARY_NAME);
        } catch (UnsatisfiedLinkError e) {
            Log.e("UnsatisfiedLinkError",
                    "Failed to load native OpenVINO libraries\n" + e.toString());
            System.exit(1);
        }

        modelDir = this.getExternalFilesDir(Environment.DIRECTORY_DOCUMENTS).getAbsolutePath();
        if(checkSelfPermission(Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
            requestPermissions(new String[]{Manifest.permission.CAMERA}, 0);
        } else {
            processNetwork();
        }
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, String[] permissions, int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        if (grantResults.length > 0 && grantResults[0] != PackageManager.PERMISSION_GRANTED) {
            Log.e("PermissionError", "The application can't work without camera permissions");
            System.exit(1);
        }
        processNetwork();
    }

    @Override
    public void onResume() {
        super.onResume();
        mOpenCvCameraView.enableView();
    }

    @Override
    public void onCameraViewStarted(int width, int height) {}

    @Override
    public void onCameraViewStopped() {}

    @Override
    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
        Mat frame = inputFrame.rgba();
        Mat frameBGR = new Mat();
        Imgproc.cvtColor(frame, frameBGR, Imgproc.COLOR_RGBA2RGB);
        int[] dimsArr = {1, frameBGR.channels(), frameBGR.height(), frameBGR.width()};
        TensorDesc tDesc = new TensorDesc(Precision.U8, dimsArr, Layout.NHWC);
        Blob imgBlob = new Blob(tDesc, frameBGR.dataAddr());
        inferRequest.SetBlob(inputName, imgBlob);
        inferRequest.Infer();
        Blob outputBlob = inferRequest.GetBlob(outputName);
        float[] scores = new float[outputBlob.size()];
        outputBlob.rmap().get(scores);
        int numDetections = outputBlob.size() / 7;
        int confidentDetections = 0;
        for (int i = 0; i < numDetections; i++) {
            float confidence = scores[i * 7 + 2];
            if (confidence > CONFIDENCE_THRESHOLD) {
                float xmin = scores[i * 7 + 3] * frameBGR.cols();
                float ymin = scores[i * 7 + 4] * frameBGR.rows();
                float xmax = scores[i * 7 + 5] * frameBGR.cols();
                float ymax = scores[i * 7 + 6] * frameBGR.rows();
                Imgproc.rectangle(frame, new Point(xmin, ymin), new Point(xmax, ymax), new Scalar(0, 0, 255), 6);
                confidentDetections++;
            }
        }
        Imgproc.putText(frame, String.valueOf(confidentDetections), new Point(10, 40),
                Imgproc.FONT_HERSHEY_COMPLEX, 1.8, new Scalar(0, 255, 0), 6);
        return frame;
    }

    private CameraBridgeViewBase mOpenCvCameraView;
    private InferRequest inferRequest;
    private String inputName;
    private String outputName;
    private String modelDir;
    public static final double CONFIDENCE_THRESHOLD = 0.5;
    public static final String OPENCV_LIBRARY_NAME = "opencv_java4";
    public static final String PLUGINS_XML = "plugins.xml";
    public static final String MODEL_XML = "face-detection-adas-0001.xml";
    public static final String MODEL_BIN = "face-detection-adas-0001.bin";
    public static final String DEVICE_NAME = "CPU";
}

• Download model “face-detection-adas-0001” from Open Model Zoo:

git clone --depth 1 https://github.com/openvinotoolkit/open_model_zoo
cd open_model_zoo/tools/downloader
python3 -m pip install -r requirements.in
python3 downloader.py --name face-detection-adas-0001

• Create assets folder in the project: Select “File – New – Folder – Assets Folder” and copy following files to this folder:

~/android_ov/openvino/bin/aarch64/Release/lib/plugins.xml
~/android_ov /open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml
~/android_ov/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.bin

• Build the project and run the application on your Android device. To do that, connect Android device to PC, select it device in drop-down menu on the toolbar and press the “Run” button. Pay attention! For Samsung devices please add permissions for transfer data with help USB.
• On the first run you need to grant camera and external storage read permissions to the application. If you see a black screen or camera error after providing the permissions — try to run the application again.