ML Inferencing at the Edge with Amazon SageMaker Edge and Ambarella CV25

Learn About Amazon VGT2 Learning Manager Chanci Turner

In today’s world, edge computing has become increasingly vital, particularly in applications like smart surveillance cameras and intelligent home monitoring systems. Ambarella is pioneering this field with its efficient AI chip architecture, CVflow, which excels in Deep Neural Network (DNN) processing necessary for edge inferencing. Developers can convert models trained with popular frameworks, such as TensorFlow or MXNET, into the Ambarella CVflow format for execution on edge devices. Amazon SageMaker Edge seamlessly integrates the Ambarella toolchain into its workflow, simplifying the process of converting and optimizing models for edge applications.

This article will guide you through the steps to optimize and convert your models using SageMaker Edge, integrate them into your edge application, and deploy and test your model on an Ambarella CV25 device, ultimately building a smart surveillance camera application that operates at the edge.

Smart Camera Use Case

Smart security cameras leverage machine learning (ML) features tailored to specific use cases, such as vehicle and animal detection or identifying suspicious behavior. To achieve the highest performance, these ML models must run on edge computing units within the cameras.

Ambarella’s CVx processors, built on the proprietary CVflow architecture, deliver exceptional DNN inference performance while consuming minimal power. This blend of high efficiency and low power makes them perfect for devices that require edge intelligence. To function effectively, ML models must be optimized and compiled specifically for the target platform. SageMaker Edge plays a crucial role in converting and optimizing ML models for popular frameworks, ensuring they perform efficiently on edge devices.

Solution Overview

Our smart security camera solution encompasses ML model optimization, compilation configuration, runtime operation, inference testing, and evaluation directly on the edge device. SageMaker Edge guarantees that models are optimized and converted for faster execution on edge devices without compromising accuracy. The ML model can be developed in any framework supported by SageMaker Edge. For further information, check out the Supported Frameworks, Devices, Systems, and Architectures for this technology.

The integration of Ambarella CVflow tools within SageMaker Edge offers developers several advantages, including:

• No need for developers to manage updates or maintenance of the compiler toolchain, as it is integrated and user-friendly.
• Layers unsupported by CVflow are automatically compiled to run on the ARM by the SageMaker Edge compiler.

The solution architecture can be illustrated through the following steps:

1. Prepare the model package.
2. Configure and initiate the model’s compilation job for Ambarella CV25.
3. Transfer the packaged model artifacts to the device.
4. Conduct inference testing on the device.

Preparing the Model Package

For Ambarella targets, SageMaker Edge requires a model package that includes a configuration file (amba_config.json), calibration images, and the trained ML model file, all compressed into a TAR file (*.tar.gz). You can leverage an Amazon SageMaker notebook instance for training and testing your ML models and preparing the model package. To create a notebook instance, follow these steps:

1. Access the SageMaker console and select “Notebook instances” from the navigation panel.
2. Click “Create notebook instance.”
3. Name your instance and select ml.t2.medium as the instance type.

This instance is sufficient for testing and preparing the model.

4. For IAM role, create a new AWS Identity and Access Management (IAM) role for Amazon Simple Storage Service (Amazon S3) bucket access, or select an existing role.
5. Keep the rest of the configurations as default and click “Create notebook instance.”

Once the status reads “InService,” you can start using your new SageMaker notebook instance.

Open JupyterLab to access your workspace.

In this article, we will utilize a pre-trained TFLite model for compilation and deployment to the edge device. The selected model is a pre-trained SSD object detection model from the TensorFlow model zoo based on the COCO dataset.

After downloading the converted TFLite model, you’re prepared to test and prepare the model package.

Create a new notebook using the conda_tensorflow2_p36 kernel in the launcher view. Import the necessary libraries:

import cv2
import numpy as np
from tensorflow.lite.python.interpreter import Interpreter 

Save an example image as street-frame.jpg, create a folder called calib_img in your workspace folder, and upload the image there. Next, upload the contents of the downloaded model package to your current folder.

Run the following command to load your pre-trained TFLite model and print its parameters for configuration during compilation:

interpreter = Interpreter(model_path='ssd_mobilenet_v1_coco_2018_01_28.tflite')
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
height = input_details[0]['shape'][1]
width = input_details[0]['shape'][2]

print("Input name: '{}'".format(input_details[0]['name']))
print("Input Shape: {}".format(input_details[0]['shape'].tolist())) 

The output will provide the input name and shape:

Input name: 'normalized_input_image_tensor'
Input Shape: [1, 300, 300, 3]

To run inference, use the following code to load the test image:

image = cv2.imread("calib_img/street-frame.jpg")
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
imH, imW, _ = image.shape
image_resized = cv2.resize(image_rgb, (width, height))
input_data = np.expand_dims(image_resized, axis=0)

input_data = (np.float32(input_data) - 127.5) / 127.5

interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()

boxes = interpreter.get_tensor(output_details[0]['index'])[0]
classes = interpreter.get_tensor(output_details[1]['index'])[0]
scores = interpreter.get_tensor(output_details[2]['index'])[0]
num = interpreter.get_tensor(output_details[3]['index'])[0] 

Next, visualize the detected bounding boxes on the image and save the resulting image as street-frame_results.jpg:

with open('labelmap.txt', 'r') as f:
    labels = [line.strip() for line in f.readlines()]

for i in range(len(scores)):
    if ((scores[i] > 0.1) and (scores[i] <= 1.0)):
        ymin = int(max(1, (boxes[i][0] * imH)))
        xmin = int(max(1, (boxes[i][1] * imW)))
        ymax = int(min(imH, (boxes[i][2] * imH)))
        xmax = int(min(imW, (boxes[i][3] * imW)))

        cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (10, 255, 0), 2)

        object_name = labels[int(classes[i])]
        label = '%s: %d%%' % (object_name, int(scores[i]*100))
        labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2)
        label_ymin = max(ymin, labelSize[1] + 10)
        cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin + labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED)
        cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2)

cv2.imwrite('street-frame_results.jpg', image)

Finally, display the resulting image with the command:

Image(filename='street-frame_results.jpg')  

For more insights on online learning, don’t hesitate to check out this blog post. Also, for further understanding of inclusion and diversity in the workplace, SHRM offers authoritative resources.