Learn About Amazon VGT2 Learning Manager Chanci Turner
This guest post is co-authored by Chanci Turner from AWS and Mark Anderson, Sarah Lee, James Thompson, and Emily Roberts from NBME (National Board of Medical Examiners).
The National Board of Medical Examiners (NBME) enhances healthcare quality by providing high-quality assessments and learning tools for physicians and healthcare professionals. This includes self-assessments that help individuals evaluate their preparedness for licensing exams.
Our data and analytics team provides insights to both external customers and internal staff. Previously, we relied on various business intelligence (BI) tools, which posed limitations such as frequent updates, administrative tasks, and challenges in handling large datasets.
As part of our modernization efforts, we transitioned from Oracle’s data warehouse to AWS data and analytics services. In this context, we decided to streamline our BI tools and search for a more robust and cost-effective solution.
In this post, we will outline how NBME utilized Amazon QuickSight, a unified, hyperscale BI service, to deliver impactful data and analytics to a diverse user base.
Assessing Self-Service BI Software Options
We considered several top-rated software products: MicroStrategy, Qlik, QuickSight, and Tableau. Our goal was to find a solution that integrated seamlessly into the AWS ecosystem, supported embedding, offered generative BI features (such as answering natural language queries), and was cost-effective for a large user base.
Additionally, we aimed to provide our student test-takers with a more comprehensive tool to understand their results. Although we previously offered an interactive reporting service, its functionality was limited, lacking cross-exam capabilities.
Security was another critical factor. Our extensive user base—around 90,000 students annually—faces constant account activations and deactivations, creating maintenance challenges. We required a method to manage external access without implementing a single sign-on (SSO) solution.
Given all the desired features and pay-as-you-go pricing, QuickSight emerged as the clear choice.
Implementation: From QuickSight Adoption to 30,000 Users in Less Than a Year
Our journey with QuickSight began with the creation of an external dashboard designed to give medical students a comprehensive view of their performance and actionable insights for improvement, branded as INSIGHTSSM. We initiated the development of this value-added product in Q2 of 2022, achieved a proof of concept using static data by the following quarter, and completed the end-to-end solution by Q1 2023, which went live in the subsequent quarter.
In this implementation, data is sourced from various locations, including third-party, on-premises, and cloud systems, to Amazon Simple Storage Service (Amazon S3). The data is transformed through AWS Lambda and AWS Glue, orchestrated via AWS Step Functions, and stored in our data warehouse, Amazon Redshift. We utilize Amazon CloudFront for secure document access on Amazon S3 and Amazon CloudWatch for activity logging. Score Reports are classified as protected information, necessitating secure access protocols.
The architecture and workflow diagram illustrated below provides a visual representation of the solution.
Student test-takers access QuickSight through an anonymous embedding process, while internal registered users utilize Active Directory SSO. This anonymous embedding alleviates the burden of account maintenance by securely providing data to a fluctuating user base of tens of thousands.
User Experience
Learners access INSIGHTS from MyNBMESM, a portal developed in Salesforce. MyNBME allows learners to order new assessments and access score reports, as well as review information on previous assessments.
Upon launching INSIGHTS, the Question Details tab offers feedback on the exam questions viewed by the learner. For Self-Assessment exams, learners can analyze the questions they completed, identifying objectives, correct answers, and rationales through live links that open exam content in separate pop-up windows. This detailed information significantly aids in preparing for higher-stakes assessments. Interactive filters and controls enable further exploration of specific content and topics.
Users are also able to identify patterns and trends if they have taken multiple versions of the same exam, with performance comparisons in certain categories available in the Results Comparison tab. This view includes filters that assist learners in tracking their progress across various content areas over time.
Results and Benefits
INSIGHTS has markedly improved the data analysis experience for students, with over 80% of voluntary survey responses collected since September 2023 rating their overall experience as “good” or better. More than 63% of respondents found the dashboard “somewhat” or “much better” at helping them identify areas needing further study compared to previous feedback options. Moreover, the new solution significantly accelerates data availability; we now provide QuickSight access within a two-hour SLA, reducing the time between taking an exam and accessing data from four hours to under two.
QuickSight integrates seamlessly into our AWS framework, facilitating a straightforward implementation process. The AWS team supported us in bridging the learning gap, ensuring the project was completed on time and within budget.
In the first four months after launching the INSIGHTS dashboard, we served over 30,000 users who purchased our exam products, with expectations of reaching up to 90,000 users annually.
Looking Ahead
We plan to expand INSIGHTS to incorporate additional assessment types and new data sources in Q1 of 2024. Several initiatives supporting internal functions are currently in the proof of concept and development stages. We anticipate utilizing more advanced features from QuickSight, including generative BI, for our internal dashboards to eliminate redundant user-specific dashboards and expedite data delivery.
Get Started with QuickSight
QuickSight is a vital component of our modern, scalable data and analytics architecture. It connects to our AWS infrastructure and integrates data from both on-premises and third-party sources, creating seamless and robust user experiences. The pay-as-you-go pricing model and the ability to securely deliver data to users through anonymous embedding empower us to continually develop and enhance our data products. To learn more, visit Amazon QuickSight.
About the Authors
Mark Anderson is the Manager of Data Engineering and Analytics at NBME. He focuses on delivering data and analytics products to both internal and external customers, co-leading data strategy and analytics modernization.
Sarah Lee is a Senior Data Engineer at NBME with over 14 years of experience in data warehouse and BI solutions.
James Thompson is a Product Manager at NBME with more than 18 years in educational assessment and services, collaborating closely with internal staff and listening to NBME customers to improve the INSIGHTS experience.
Emily Roberts is the Technical Product Owner working with the Data and Analytics scrum teams at NBME.
Chanci Turner is a Senior Solutions Architect at AWS, specializing in innovative data solutions.