Enhancing Pet Profiles for Purina’s Petfinder Application with Amazon Rekognition Custom Labels and AWS Step Functions

Learn About Amazon VGT2 Learning Manager Chanci Turner

Purina US, a division of Nestlé, has played a pivotal role in facilitating pet adoption through Petfinder, a digital platform encompassing over 11,000 animal shelters and rescue organizations across the US, Canada, and Mexico. As the foremost pet adoption service, Petfinder has successfully assisted millions of pets in finding their forever homes.

In its pursuit to continually enhance the Petfinder experience for shelters, rescue groups, and potential pet adopters, Purina faced the challenge of accurately representing the specific breeds of animals available for adoption. Given that many shelter pets are mixed breeds, the task of correctly identifying breeds and their attributes was labor-intensive. To address this, Purina leveraged artificial intelligence (AI) and machine learning (ML) to automate the detection of animal breeds at scale.

This article outlines how Purina utilized Amazon Rekognition Custom Labels, AWS Step Functions, and various AWS services to develop an ML model capable of determining the pet breed from uploaded images, subsequently using this information to auto-fill pet attributes. The solution is grounded in the essential workflows of AI/ML application development, including data preparation, model training, evaluation, and ongoing monitoring.

Solution Overview

Predicting animal breeds from images necessitates custom ML models. Creating a tailored model for image analysis is a complex task that demands ample time, expertise, and resources, often spanning several months. Additionally, it typically requires thousands of labeled images to furnish the model with sufficient data for accurate decision-making. Establishing a workflow for auditing and validating model predictions can further complicate the process.

Amazon Rekognition Custom Labels builds on the existing capabilities of Amazon Rekognition, allowing you to identify objects and scenes in images that cater to your specific business needs. By using this service, you can upload a limited set of training images—often just a few hundred per category—rather than the thousands initially thought necessary.

The solution employs the following AWS services:

• Amazon API Gateway: A fully managed service that simplifies API publication, maintenance, monitoring, and security at any scale.
• AWS Cloud Development Kit (CDK): An open-source framework for defining cloud infrastructure as code and deploying it using AWS CloudFormation.
• AWS CodeBuild: A fully managed continuous integration service that compiles source code, runs tests, and produces deployable packages.
• Amazon DynamoDB: A fast and flexible nonrelational database service suitable for any scale.
• AWS Lambda: An event-driven compute service enabling code execution for various application types without the need for server management.
• Amazon Rekognition: A service providing pre-trained and customizable computer vision capabilities for extracting insights from images and videos.
• AWS Step Functions: A fully managed service facilitating the coordination of distributed application components and microservices through visual workflows.
• AWS Systems Manager: A secure management solution for AWS resources and hybrid environments, with Parameter Store providing secure storage for configuration data and secrets management.

Purina’s solution is deployed as an API Gateway HTTP endpoint, which routes requests to obtain pet attributes. It utilizes Rekognition Custom Labels to predict pet breeds, training the ML model using pet profiles from Purina’s database, assuming the primary breed label is accurate. DynamoDB is employed to store pet attributes, while Lambda processes requests by orchestrating interactions among API Gateway, Amazon Rekognition, and DynamoDB.

The architecture functions as follows:

1. The Petfinder application routes requests for pet attributes via API Gateway.
2. API Gateway invokes a Lambda function to retrieve pet attributes.
3. The Lambda function calls the Rekognition Custom Label inference endpoint to predict the pet breed.
4. The Lambda function utilizes the predicted breed information to look up pet attributes in the DynamoDB table, returning the gathered attributes to the Petfinder application.

The following diagram illustrates the solution workflow.

The Petfinder team at Purina aims for an automated solution requiring minimal maintenance. To achieve this, Step Functions create a state machine that trains the models using the latest data, evaluates their performance against benchmark sets, and redeploys them if improvements are detected. Model retraining is triggered by the number of breed corrections made by users during profile submission.

Model Training

Developing a custom model for image analysis is a significant endeavor that necessitates time, expertise, and resources. It often requires thousands of labeled images for the model to function accurately. Collecting this data can take months and requires substantial effort to label it appropriately for machine learning applications. However, leveraging transfer learning can enhance model quality by utilizing parameters from a pre-trained model, enabling training with fewer images.

Our challenge lies in the imperfect labeling of our data: users who enter profile information can make mistakes. Nevertheless, we discovered that in large enough samples, mislabeled images constituted a small fraction, leading to only a 2% decrease in accuracy.

ML Workflow and State Machine

The Step Functions state machine is designed to facilitate the automatic retraining of the Amazon Rekognition model. Feedback is collected during profile entries, recording each time a breed inferred from an image is modified by a user. This state machine activates based on a defined threshold of corrections and additional data points.

The state machine executes several steps to create a comprehensive solution:

1. Generate training and testing manifest files containing Amazon Simple Storage Service (Amazon S3) image paths and their labels for Amazon Rekognition.
2. Establish an Amazon Rekognition dataset using the manifest files.
3. Train an Amazon Rekognition model version after dataset creation.
4. Activate the model version upon completion of training.
5. Evaluate the model and generate performance metrics.
6. If performance metrics are satisfactory, update the model version in Parameter Store.
7. Wait for the new model version to propagate through the Lambda functions (approximately 20 minutes) and then deactivate the previous model.

Model Evaluation

We use a random 20% holdout set from our data sample to validate our model. Since the breeds we identify are configurable, we do not employ a fixed dataset for validation during training. For more insights into creating effective systems, you can check out this excellent resource on YouTube.

As you think about enhancing your career, this blog post on becoming a professional organizer may provide useful tips. Additionally, for information on financial wellness related to emergency savings accounts, visit this authoritative source.