HOME

Fine-Tuning Instructions for FLAN T5 XL with Amazon SageMaker Jumpstart

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

Generative AI is experiencing remarkable advancements, with an influx of highly capable foundation models being introduced regularly. Among these, large language models (LLMs) stand out as a prominent category. LLMs consist of billions of parameters trained on vast datasets, encompassing hundreds of billions or even a trillion tokens. These models excel in a variety of text-centric tasks, ranging from question answering to sentiment analysis.

The strength of LLMs lies in their ability to learn and generalize from extensive and diverse training datasets. Their initial training involves various objectives, whether supervised, unsupervised, or hybrid. One of the most prevalent unsupervised objectives is text completion, where the model predicts what follows a given text segment (for instance, predicting the next sentence). Supervised training can also be applied using labeled data to accomplish specific tasks (e.g., determining if a movie review is positive, negative, or neutral). However, the tasks for which models are trained may not align with the specific needs of users.

To enhance the performance of a pre-trained LLM on a specific task, we can refine the model using examples of the desired task through a process called instruction fine-tuning. This technique employs a set of labeled examples structured as {prompt, response} pairs to further train the pre-trained model, enabling it to predict responses accurately based on given prompts. This process adjusts the model’s weights.

This article outlines the steps for instruction fine-tuning an LLM, specifically FLAN T5 XL, using Amazon SageMaker Jumpstart. We demonstrate this process through both the Jumpstart user interface and a notebook in Amazon SageMaker Studio. The accompanying notebook is available in the amazon-sagemaker-examples GitHub repository.

Overview of the Solution

The focus of this post is to generate questions related to a provided text prompt that cannot be answered based on the information contained within that text. This task is valuable for identifying gaps in information or determining whether a query requires additional context.

FLAN T5 models are instruction fine-tuned across a wide array of tasks to improve their zero-shot performance on common tasks. Further instruction fine-tuning for a specific customer task can significantly boost these models’ accuracy, particularly if the task hasn’t been previously used to train a FLAN T5 model, which is the case for our example.

In our scenario, we aim to formulate relevant but unanswerable questions. To achieve this, we utilize a subset of version 2 of the Stanford Question Answering Dataset (SQuAD2.0). This dataset comprises questions created by human annotators based on Wikipedia articles, including roughly 50,000 unanswerable questions—plausible questions that cannot be answered directly from the content of the articles. Our data is organized in a JSON Lines format, with each entry containing a context and a question.

Prerequisites

To begin, you will need an AWS account where you can access Studio. If you don’t have a user profile for Studio, you will need to create one.

Fine-Tuning FLAN-T5 with the Jumpstart UI

To fine-tune the model using the Jumpstart UI, follow these steps:

  1. Open Studio in the SageMaker console.
  2. Navigate to SageMaker Jumpstart in the left pane, then select Models, notebooks, solutions.

You will see a list of foundation models, including FLAN T5 XL, which is designated as fine-tunable.

  1. Click on View model.
  2. Under Data source, provide the path to your training data. The source for the data used in this article is preset.
  3. You may retain the default settings for deployment configuration, security, and hyperparameters; however, you should increase the number of epochs to a minimum of three for optimal results.
  4. Click Train to initiate the model training.

You can monitor the training job status through the UI.

  1. Upon completion of training (which takes about 53 minutes in our case), click Deploy to launch the fine-tuned model.

After the endpoint is established (which takes a few minutes), you can open a notebook and start utilizing your fine-tuned model.

Fine-Tuning FLAN-T5 with a Python Notebook

Our example notebook illustrates how to programmatically fine-tune and deploy a FLAN T5 XL model using Jumpstart and SageMaker. It can be executed in Studio or locally.

In this section, we will first go through some general setup. Following that, you will fine-tune the model with the SQuADv2 datasets. Next, you will deploy both the pre-trained and fine-tuned versions of the model behind a SageMaker endpoint. Finally, you can query these endpoints and compare the output quality of the pre-trained and fine-tuned models. You will observe that the fine-tuned model produces significantly higher quality output.

Setting Up Prerequisites

Start by installing and upgrading the necessary packages. Restart the kernel after running the following commands:

!pip install nest-asyncio==1.5.5 --quiet
!pip install ipywidgets==8.0.4 --quiet
!pip install --upgrade sagemaker --quiet

Next, obtain the execution role linked to the current notebook instance:

import boto3
import sagemaker

# Get current region, role, and default bucket
aws_region = boto3.Session().region_name
aws_role = sagemaker.session.Session().get_caller_identity_arn()
output_bucket = sagemaker.Session().default_bucket()

# This will be useful for printing
newline, bold, unbold = "n", "33[1m", "33[0m"
print(f"{bold}aws_region:{unbold} {aws_region}")
print(f"{bold}aws_role:{unbold} {aws_role}")
print(f"{bold}output_bucket:{unbold} {output_bucket}")

You can create a convenient drop-down menu to display the model sizes available for fine-tuning:

import IPython
from ipywidgets import Dropdown
from sagemaker.jumpstart.filters import And
from sagemaker.jumpstart.notebook_utils import list_jumpstart_models

# Default model choice
model_id = "huggingface-text2text-flan-t5-xl"

# Identify FLAN T5 models that support fine-tuning
filter_value = And(
    "task == text2text", "framework == huggingface", "training_supported == true"
)
model_list = [m for m in list_jumpstart_models(filter=filter_value) if "flan-t5" in m]

# Display the model IDs in a dropdown, for user to select
dropdown = Dropdown(
    value=model_id,
    options=model_list,
    description="FLAN T5 models available for fine-tuning:",
    style={"description_width": "initial"},
    layout={"width": "max-content"},
)
display(IPython.display.Markdown("### Select a pre-trained model from the dropdown below"))
display(dropdown)

Jumpstart automatically retrieves the appropriate training and inference instance types for the selected model.

For more insights into professional communication, check out this blog post on professional emails. Additionally, if you’re interested in immigration topics, you can read about the recent updates on filing fees from USCIS, an authority on this matter. Furthermore, for those looking for career opportunities, this role as a Learning Trainer in Los Angeles could be an excellent resource.

Related Topics:

HOME