Neuron SDK Demo on I-Pi SMARC LEC-1200

Overview

NNStreamer is a set of Gstreamer plugins that allow Gstreamer developers to adopt neural network models, and neural network developers to manage neural network pipelines with their filters in an easy and efficient way.

NNStreamer provides the new Gstreamer stream data type and a set of Gstreamer elements (plugins) to construct media stream pipeline with neural network models. It is well documented through its online document site and it supports well-known neural network frameworks including Tensorflow, Tensorflow-lite, Caffe2, PyTorch, OpenVINO and ARMNN.

Users may include custom C functions, C++ objects, or Python objects as well as such frameworks as neural network filters of a pipeline in run-time and also add and integrate support for such frameworks or hardware AI accelerators in run-time, which may exist as independent plugin binaries.

For more details, visit NNStreamer — IoT Yocto documentation (mediatek.gitlab.io)

Prerequisites

Hardware

Module  : LEC-MTK-1200
Carrier : I-Pi SMARC Plus A3

Software

OS             : adlink-lec-1200-yocto-kirkstone_V3_R1_240201
Kernal version : 5.15-lecmtki1200-3v1

Prepare the Target Board

Download the image, flash and halt at the U-Boot prompt and execute the following commands in the Ubuntu terminal.

Note: Follow this link: 1200-docs – I-Pi SMARC (ipi.wiki) for flashing Yocto based image on the target. 

$ setenv boot_conf "#conf-mediatek_lec-mtk-i1200-ufs.dtb#conf-gpu-mali.dtbo#conf-video.dtbo#conf-apusys.dtbo" 
$ saveenv 
$ reset

NNStreamer Examples

Yocto provides following examples in python and /usr/bin/nnstreamer-demo/

Python Script Category
nnstreamer_example_face_detection_uvc.py Face detection
nnstreamer_example_pose_estimation_uvc.py Pose estimation
nnstreamer_example_object_detection_uvc.py Object detection
nnstreamer_example_image_classification_uvc.py Image classification

To run these NNStreamer_examples, you will require a V4L2-compatible device. A USB webcam can serve as a V4L2 video device, interfacing through Gstreamer. To identify the USB camera, refer to the command below:

Note: Connect the USB camera to the target board.

ls -l /sys/class/video4linux/ | grep xhci
NeuroPilot SDK Demo instruction on I-Pi SMARC 1200

From the above stated command, we can find that /dev/video5 is the camera node. Use the following command to verify the detected camera node generated upon connecting the USB camera.

v4l2-ctl --list-formats-ext -d /dev/video5
NeuroPilot SDK Demo instruction on I-Pi SMARC 1200

Below listed options are supported:

  • --engine:

Choose one from the backends supported by the platform to use. It could be neuronsdk, tflite or armnn .

Take nnstreamer_example_pose_estimation_uvc.py as an example:

  1. --engine neuronsdk :

    python3 nnstreamer_example_pose_estimation_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

  2. --engine tflite :

    python3 nnstreamer_example_pose_estimation_uvc.py --engine tflite --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

  3. --engine armnn :

    python3 nnstreamer_example_pose_estimation_uvc.py --engine armnn --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

  • --cam: Input a camera node id.

  • --performance: Set performance mode for your platform.

Face Detection

Face detection example on I-Pi SMARC 1200 NeuroPilot

  • Python script: /usr/bin/nnstreamer-demo/nnstreamer_example_face_detection.py

  • Run the script:

    • Execute on MDLA by neuronsdk:

      python3 nnstreamer_example_face_detection_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on tflite:

      python3 nnstreamer_example_face_detection_uvc.py --engine tflite --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on GPU by ArmNN:

      python3 nnstreamer_example_face_detection_uvc.py --engine armnn --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

Pose Estimation

Pose estimation example on I-Pi SMARC 1200 NeuroPilot

  • Python script: /usr/bin/nnstreamer-demo/nnstreamer_example_pose_estimation_uvc.py

  • Run the script:

    • Execute on MDLA by neuronsdk:

      python3 nnstreamer_example_pose_estimation_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on tflite:

      python3 nnstreamer_example_pose_estimation_uvc.py --engine tflite --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on GPU by ArmNN:

      python3 nnstreamer_example_pose_estimation_uvc.py --engine armnn --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

Object Detection

Object detetction example on I-Pi SMARC 1200 NeuroPilot

  • Python script: /usr/bin/nnstreamer-demo/nnstreamer_example_object_detection_uvc.py

  • Run the script:

    • Execute on MDLA by neuronsdk:

      python3 nnstreamer_example_object_detection_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on tflite:

      python3 nnstreamer_example_object_detection_uvc.py --engine tflite --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on GPU by ArmNN:

      python3 nnstreamer_example_object_detection_uvc.py --engine armnn --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

Image Classification

Image classification example on I-Pi SMARC 1200 NeuroPilot

  • Python script: /usr/bin/nnstreamer-demo/nnstreamer_example_image_classification_uvc.py

  • Run the script:

    • Execute on MDLA by neuronsdk:

      python3 nnstreamer_example_image_classification_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on tflite:

      python3 nnstreamer_example_image_classification_uvc.py --engine tflite --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    • Execute on GPU by ArmNN:

      python3 nnstreamer_example_image_classification_uvc.py --engine armnn --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

Video Testing on Neuron SDK Demo

  • To download the file, click while holding down the Ctrl key on your keyboard on the object.zip link. Then, choose the “extract” option and select a connected USB flash drive to extract the “object” file onto the USB flash drive.

    Extract file for video testing on Neuron SDK Demo instruction. Extract file for video testing on Neuron SDK Demo instruction.

  • Insert the USB drive into the LEC_MTKi1200 machine, and then copy the files object.sh and traffic.mp4 into (/usr/bin/nnstreamer-demo) directory.

  • Finally, execute the command below on terminal:

    cd /usr/bin/nnstreamer-demo

    python3 nnstreamer_example_object_detection_uvc.py --engine neuronsdk --cam 5 --width 640 --height 480 --performance G1200 --fullscreen 1

    ./object.sh traffic.mp4

    Example of Neuron SDK Demo.
Prev