Chanci Turner

Extracting Valuable Insights from Custom Jira Projects with Amazon QuickSight

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

Extracting valuable insights from custom Jira projects is essential for organizations aiming to refine their processes and foster innovation. However, navigating the complexities of analyzing unique Jira configurations and data structures can be daunting. With Amazon QuickSight’s generative business intelligence (BI) capabilities, companies can transform the way they interact with their Jira data. The integration of QuickSight with Jira empowers users to leverage natural language queries, creating visuals and revealing insights that may otherwise remain obscured. By merging the adaptability of custom Jira projects with the user-friendly interface of QuickSight generative BI, organizations can effortlessly uncover significant insights.

In this article, we will delve into how QuickSight’s generative BI can assist you in extracting meaningful insights from your unique Jira projects. We will present practical use cases, highlight the technology’s potential, and offer guidance on implementing this solution in your environment.

Solution Overview

This solution seamlessly integrates Jira with QuickSight to establish a robust automated pipeline for visualizing project management data. Utilizing AWS services such as AWS Lambda, Amazon Simple Storage Service (Amazon S3), AWS Glue, Amazon Athena, and QuickSight, this architecture efficiently extracts data from Jira, processes it, and generates real-time visualizations that provide actionable insights into project advancement and team performance. The generative BI features in QuickSight allow users to pose natural language questions about their Jira data, enabling both technical and non-technical users to swiftly discover insights.

This customized method offers enhanced flexibility compared to Amazon AppFlow, which typically limits extraction to standard fields and might not capture custom Jira fields or hierarchical relationships. By orchestrating the data flow through Lambda, Amazon S3, AWS Glue, Athena, and QuickSight, the solution delivers a comprehensive and tailored dataset—equipping teams with timely actionable insights for improved project visibility and data-driven decision-making.

The accompanying architecture diagram illustrates the flow of each stage in the data pipeline, from triggering a daily event for data extraction from Jira to updating dashboards in QuickSight. This automation ensures teams have access to the latest project data without requiring manual updates.

Workflow Steps

  1. Amazon EventBridge is configured to trigger the data ingestion process daily. This trigger can be customized to fit the refresh frequency required for your use case. EventBridge activates Lambda for data extraction from Jira.
  2. Lambda retrieves data from Jira using its API when prompted by EventBridge. The Lambda function collects relevant project data from Jira based on predetermined parameters.
  3. The raw data from Jira is stored in the S3 raw data bucket. This bucket serves as a centralized repository for all incoming Jira data.
  4. AWS Glue processes and transforms the raw data stored in Amazon S3. It performs data cleaning, structuring, and cataloging, preparing the data for further analysis.
  5. The transformed data is saved into the S3 processed data bucket. This bucket contains structured and cleaned data optimized for querying and analysis.
  6. Athena is set up to query the transformed data in the processed S3 bucket. It enables SQL-based querying, and custom views can be created to support hierarchical structures like themes, epics, and tasks.
  7. QuickSight connects to Athena to visualize the processed Jira data. QuickSight dashboards allow users to explore data through charts and graphs, drill down into hierarchies, and utilize Amazon Q for natural language querying, facilitating real-time insights.

The following sections will guide you through the steps necessary for extracting, transforming, and visualizing Jira data using AWS services. By following these steps, you can automate data retrieval from Jira, transform it for analysis, and create meaningful visualizations in QuickSight.

Prerequisites

To enable the integration between QuickSight and Jira, you need an AWS account with access to the following services:

  • AWS Account – An active AWS account is required. If you don’t have one yet, you can sign up on the AWS console.
  • QuickSight subscription – Obtain a subscription to QuickSight.
  • Grant QuickSight access to S3 and Athena: Authorizing connections to Amazon Athena.
  • Amazon Athena
  • Amazon EventBridge
  • AWS Glue
  • AWS Lambda
  • Amazon QuickSight
  • Amazon S3

For Jira access, you need:

  • A valid Jira account (compatible with Jira Cloud and Jira Server versions 8.0 and above). Jira Cloud and Jira Server provide robust REST APIs, but they differ mainly in authentication and endpoint structure. Jira Cloud uses cloud-based endpoints and requires API tokens for secure access, while Jira Server may use alternative authentication methods and can have subtle variations in API endpoints. Review the specific API documentation for your Jira environment for a seamless integration.
  • A Jira API token with appropriate permissions.
  • API access to relevant Jira projects.

Ensure you have the necessary AWS Identity and Access Management (IAM) roles and policies set up with permissions for the following actions:

  • Create and manage Lambda functions:
    • lambda:CreateFunction
    • lambda:UpdateFunctionCode
    • lambda:InvokeFunction
    • lambda:DeleteFunction
  • Read from and write to S3 buckets:
    • s3:GetObject
    • s3:PutObject
    • s3:ListBucket
    • s3:DeleteObject
  • Create and run AWS Glue jobs:
    • glue:CreateJob
    • glue:StartJobRun
    • glue:GetJobRun
    • glue:DeleteJob
  • Execute Athena queries:
    • athena:StartQueryExecution
    • athena:GetQueryExecution
    • athena:GetQueryResults
    • athena:ListDatabases
    • athena:ListTableMetadata

Extracting Jira Data Using Lambda

The first step in this solution is to set up a Lambda function that will periodically retrieve data from Jira and store it in an S3 bucket. You have the option of integrating this code into your extract, transform, and load (ETL) script, eliminating the need for a separate Lambda function.

Complete the following steps:

  1. On the Lambda console, select Functions in the navigation pane.
  2. Choose Create function.
  3. Leave the default as Author from scratch.
  4. Enter pull-jira-data for the Function name, select Python 3.13 for Runtime, and click Create function.
  5. Use the provided Lambda function code, updating it with your API token, email, domain, and bucket name if different from Jira-data-raw.
  6. Choose Deploy.

Here’s a sample code snippet for the Lambda function:

# Import required libraries
import json
import boto3
import requests
import os
import time
from requests.auth import HTTPBasicAuth
from datetime import datetime

This process can lead to improved project visibility and data-driven decision-making. For more insights on managing your time effectively during a staycation, check out this helpful article. Additionally, for an in-depth look at job openings and turnover statistics, refer to this SHRM article from an authority on the topic. If you’re considering a career at Amazon, this resource offers excellent guidance on the onboarding process.

Related Topics:

Chanci Turner