Deployment of PyTorch Model Using NCNN for Mobile Devices — Part 2
An introductory example of deploying a pretrained PyTorch model into an Android app using NCNN for mobile devices.
Introduction
As more and more deep neural networks, like CNNs, Transformers, and Large Language Models (LLMs), generative models, etc., have been developed to increase the level of artificial intelligence (AI), it is also important to deploy the deep neural networks to mobile devices, like smart phones, AR glasses, etc., to boost the usages of the deep neural networks in our lives.
In my previous post, I gave an introductory example about deploying a pretrained PyTorch model into a C++ app using NCNN. In this post, I will talk about how to integrate the C++ app into an Android app for a mobile phone. Assuming that you have read my previous post “Deployment of PyTorch Model Using NCNN for Mobile Device — Part 1” that talks about how to generate the NCNN model from the PyTorch model and how to make an C++ app that loads the NCNN model and performs inference, I will start with the generated C++ app and the NCNN model, and integrate them into an Android app for the mobile phone. I will continue to use the image classification example discussed in my previous post.
Environment setup
Let’s first set up the environment, including installation and set up of Android Studio, NDK (Native Development Kit), OpenCV for Android, and NCNN for Android.
Android studio can be installed by following Android Studio installation page[2]. I create a new Native C++ app and select Java as the language for the image classification app. Then I install and set up necessary libraries, including OpenCV for Android and NCNN Android, and integrate the image classification C++ inference code into the app.
As shown in the figure above, the app consists of the following files and necessary libraries.
- OpenCV-android-sdk. It is used for loading input images.
- ncnn-20230223-android-vulkan. It is the NCNN library for performing inference.
- CMakeLists.txt. It is automatically generated by Android Studio. But we need to modify it to compile the app.
- native-lib.cpp. A C++ source file is called by Java code using Java Native Interface (JNI). It is the entry point of native C++ code in the app.
- inference.h and inference.cpp. They are the source and header files for the image classification inference function, and the function is called by native-lib.cpp.
- MainActivity.java. It is a Java class file in an Android app that serves as the main entry point for the app’s user interface. It calls the native functions defined in native-lib.cpp.
- activity_main.xml. It shows the user interface components for visualizing the image classification result.
- Model files. The NCNN model files are placed in the “asset” folder.
- Input files. The input image files are placed in the “drawable” folder.
To write native code in Android, we need to first install Native Development Kit (NDK). NDK is a set of tools that allows us to write and use native C or C++ code in Android applications. NDK can be installed via Android Studio by following Tools -> SDK Managers -> SDK Tools. A Settings window pops up and NDK can be installed by selecting the NDK checkbox and clicking the Apply button.
OpenCV Android can be downloaded and set up in Android by the following steps.
- Download Android release of OpenCV from https://opencv.org/releases/;
- Extract it and move the subfolder OpenCV-android-sdk under it into app/src/main/cpp in the image classification app that we are making;
- Set OpenCV_DIR in app/src/main/cpp/CMakeLists.txt.
Similiarly, NCNN Android can be downloaded and set up in Android by the following steps.
- Download ncnn-android-vulkan from https://github.com/Tencent/ncnn/releases, e.g. ncnn-20230223-android-vulkan.zip;
- Extract it into app/src/main/cpp;
- Set the ncnn_DIR path in app/src/main/cpp/CMakeLists.txt.
After installing the necessary libraries, I make inference.h and inference.cpp.
/**
* inference.h
*/
#ifndef IMAGECLASSIFICATION_INFERENCE_H
#define IMAGECLASSIFICATION_INFERENCE_H
#include <opencv2/core.hpp>
#include <android/asset_manager.h>
/**
* @brief Perform inference for classifying a given image using the NCNN model.
* @param src: input image in OpenCV Mat format.
* @param mgr: AAssetManager pointer for loading NCNN model files in assets folder.
* @return the predicted class
*/
std::string Inference(cv::Mat& src, AAssetManager* mgr);
#endif //IMAGECLASSIFICATION_INFERENCE_H/**
* inference.cpp
*/
#include <algorithm>
#include <vector>
#include <string>
#include <opencv2/imgproc.hpp>
#include "net.h"
#include "inference.h"
// Perform inference for classifying a given image using the NCNN model
std::string Inference(cv::Mat& src, AAssetManager* mgr) {
std::vector<std::string> classes = {"plane", "car", "bird", "cat",
"deer", "dog", "frog", "horse",
"ship", "truck"};
// Load model
ncnn::Net net;
int ret = net.load_param(mgr, "image_classifier_opt.param");
if (ret) {
__android_log_print(ANDROID_LOG_ERROR, "load_param_error",
"Failed to load the model parameters");
}
ret = net.load_model(mgr,"image_classifier_opt.bin");
if (ret) {
__android_log_print(ANDROID_LOG_ERROR, "load_weight_error",
"Failed to load the model weights");
}
// Convert image data to ncnn format
// opencv image in bgr, model needs rgb
ncnn::Mat input = ncnn::Mat::from_pixels(src.data,
ncnn::Mat::PIXEL_BGR2RGB,
src.cols, src.rows);
// Data preprocessing (normalization)
const float mean_vals[3] = {0.4914f*255.f,
0.4822f*255.f,
0.4465f*255.f};
const float norm_vals[3] = {1/0.2470f/255.f,
1/0.2435f/255.f,
1/0.2616f/255.f};
// In ncnn, substract_mean_normalize needs input pixels in [0, 255]
input.substract_mean_normalize(mean_vals, norm_vals);
// Inference
ncnn::Extractor extractor = net.create_extractor();
extractor.input("input", input);
ncnn::Mat C2, C3, C4, C5, output;
extractor.extract("output", output);
// Flatten
ncnn::Mat out_flatterned = output.reshape(output.w * output.h * output.c);
std::vector<float> scores;
scores.resize(out_flatterned.w);
for (int j=0; j<out_flatterned.w; j++) {
scores[j] = out_flatterned[j];
}
// Get predicted class
std::string pred_class = classes[std::max_element(scores.begin(),
scores.end()) - scores.begin()];
return pred_class;
}The inference function is called by native-lib.cpp.
/**
* native-lib.cpp
*/
#include <jni.h>
#include <string>
#include <android/bitmap.h>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <android/log.h>
#include <android/asset_manager.h>
#include <android/asset_manager_jni.h>
#include "inference.h"
/**
* @brief Convert input image from bitmap to OpenCV Mat.
* @tparam env: JNIEnv pointer.
* @tparam bitmap: input image in bitmap format.
* @param bst: output input image in OpenCV format.
* @param needUnPremultiplyAlpha: boolean variable to convert image color space
*/
void Bitmap2Mat(JNIEnv * env, jobject bitmap,
cv::Mat& dst, jboolean needUnPremultiplyAlpha) {
AndroidBitmapInfo info; // uses jnigraphics
void* pixels = 0;
CV_Assert( AndroidBitmap_getInfo(env, bitmap, &info) >= 0 );
CV_Assert( info.format == ANDROID_BITMAP_FORMAT_RGBA_8888 ||
info.format == ANDROID_BITMAP_FORMAT_RGB_565 );
CV_Assert( AndroidBitmap_lockPixels(env, bitmap, &pixels) >= 0 );
CV_Assert( pixels );
dst.create(info.height, info.width, CV_8UC4);
if( info.format == ANDROID_BITMAP_FORMAT_RGBA_8888 )
{
cv::Mat tmp(info.height, info.width, CV_8UC4, pixels);
if(needUnPremultiplyAlpha) cvtColor(tmp, dst, cv::COLOR_mRGBA2RGBA);
else tmp.copyTo(dst);
} else {
cv::Mat tmp(info.height, info.width, CV_8UC2, pixels);
cvtColor(tmp, dst, cv::COLOR_BGR5652RGBA);
}
AndroidBitmap_unlockPixels(env, bitmap);
return;
}
/**
* @brief Entry point of C++ code and MainActivity.java will call this function.
* @tparam env: JNIEnv pointer.
* @tparam bitmapIn: input image in bitmap format.
* @param assetManager: AssetManager object for loading NCNN model files in assets folder.
* @Return predicted class.
*/
extern "C" JNIEXPORT jstring JNICALL
Java_com_example_imageclassification_MainActivity_ImageClassification(
JNIEnv* env,
jobject,
jobject bitmapIn,
jobject assetManager) {
cv::Mat src;
Bitmap2Mat(env, bitmapIn, src, false); // Convert bitmap to OpenCV Mat
cv::Mat resized_src;
cv::resize(src, resized_src, cv::Size(32, 32), cv::INTER_LINEAR);
cv::cvtColor(resized_src, resized_src, cv::COLOR_RGBA2RGB);
AAssetManager* mgr = AAssetManager_fromJava(env, assetManager);
std::string pred_class = Inference(resized_src, mgr); // Image classification
return env->NewStringUTF(pred_class.c_str());
}Given all the C++ files, I make the CMakeLists.txt file.
# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html
# Sets the minimum version of CMake required to build the native library.
cmake_minimum_required(VERSION 3.22.1)
# Declares and names the project.
project("imageclassification")
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.
# OpenCV
set(OpenCV_STATIC on)
set(OpenCV_DIR ${CMAKE_SOURCE_DIR}/OpenCV--android--sdk/sdk/native/jni)
find_package(OpenCV REQUIRED)
set(ncnn_DIR ${CMAKE_SOURCE_DIR}/ncnn--20230223--android--vulkan/${ANDROID_ABI}/lib/cmake/ncnn)
find_package(ncnn REQUIRED)
add_library( # Sets the name of the library.
imageclassification
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
inference.cpp
native--lib.cpp)
# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.
find_library( # Sets the name of the path variable.
log--lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log)
find_library(jnigraphics--lib jnigraphics)
# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third--party libraries, or system libraries.
target_link_libraries( # Specifies the target library.
imageclassification
# Links the target library to the log library
# included in the NDK.
ncnn
${OpenCV_LIBS}
${jnigraphics--lib}
${log--lib})After finishing the native code, I add the code in the MainActivity.java file that calls the native code to get the classification result and display the result in the user interface.
package com.example.imageclassification;
import androidx.appcompat.app.AppCompatActivity;
import android.content.res.AssetManager;
import android.content.res.Resources;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.Bundle;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;
import com.example.imageclassification.databinding.ActivityMainBinding;
public class MainActivity extends AppCompatActivity {
// Used to load the 'imageclassification' library on application startup.
static {
System.loadLibrary("imageclassification");
}
private ActivityMainBinding binding;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
binding = ActivityMainBinding.inflate(getLayoutInflater());
setContentView(binding.getRoot());
Button btn_classify = (Button) findViewById(R.id.btn_classify);
btn_classify.setOnClickListener(new View.OnClickListener() {
public void onClick(View v) {
Resources res = getResources();
Bitmap bitmapIn =
BitmapFactory.decodeResource(res, R.drawable.car);
TextView pred = (TextView) findViewById(R.id.textView);
pred.setText(ImageClassification(bitmapIn, getAssets()));
}
});
}
/**
* A native method that is implemented by the 'imageclassification' native library,
* which is packaged with this application.
*/
public native String ImageClassification(Bitmap bitmapIn,
AssetManager assetManager);
}With all the files being created, the app can be built by doing (Build->Make Project) and run by clicking Run ’app’ button in Android Studio. You will first see the left image below, and then by clicking the “CLASSIFY” button, you will see that the car image is successfully classified into the car object.
The complete implementation of the code deployment into Android is available from the Github.
Conclusions
In this post, I discussed how to integrate the C++ code with the NCNN inference engine into Android for model deployment on the mobile phone. My discussions cover all the necessary steps to make the deployment working, including Android Studio installation, NDK installation in Android, OpenCV and NCNN setup in Android, how to integrate the C++ inference code with NCNN into Android, and how to make CMakeLists.txt to compile the C++ code in Android, and how to modify MainActivity.java to call the C++ inference code.
I hope this post together with my previous posts here and here provide you an end-to-end pipeline of deploying a pretrained PyTorch model into Android for model deployment on mobile devices. You can easily tailor the pipeline for deploying your deep learning models on mobile devices. Hope these series of posts help. Thanks for reading.
References
[1] Vaswani et al., Attention is All You Need, Dec. 2017.
[2] Android. https://developer.android.com/studio/install, 2023.
[3] Tencent, NVIDIA CUDA Convolutional Neural Network, https://github.com/tencent/ncnn, 2019.
