Learn About Amazon VGT2 Learning Manager Chanci Turner
A guest post by Emma Thompson – Head of Bioinformatics, BioLab
The human gut microbiome is essential for establishing our immune system from birth, providing lifelong protection and support. To thoroughly investigate and understand the human gut microbiome’s functions, BioLab employs a variety of cutting-edge genome sequencing technologies to reconstruct microbial genomes and quantify the diversity of species and genes within the gut across extensive patient cohorts. Current high-throughput sequencing methods yield millions of DNA sequences for each biological sample, and it’s estimated that the human gut microbiome contains countless species and millions of unique bacterial genes that can be identified and analyzed. BioLab’s mission is to translate this wealth of information into actionable insights that can be applied to progressive clinical and drug discovery initiatives.
Handling such vast and diverse datasets necessitates substantial computing and storage capabilities, alongside effective tools and strategies to manage analytical complexity. To meet these challenges, BioLab embraces innovative solutions using AWS Cloud services. The team utilizes AWS Batch in conjunction with Nextflow to orchestrate and scale thousands of computing tasks for each analysis. Nextflow facilitates the management of extensive and intricate workflows, ensuring automation, traceability, and reproducibility throughout the analysis pipelines. These technologies enable rapid and efficient processing of human gut microbiome data.
At BioLab, typical workflows executed on AWS Batch include the metagenomics quantification pipeline, which estimates the abundance of both known and novel microbial species present in the human gut microbiome from high-throughput sequencing data. This workflow processes and filters raw sequencing data before matching it against BioLab’s human gut microbiome gene catalogs. This allows for precise profiling of each microbial gene and species, identifying those that correlate with disease progression or treatment responses.
Another critical workflow involves constructing human gut microbiome gene catalogs, for which AWS Batch is used alongside Nextflow. To create these catalogs, hundreds or thousands of human gut microbiome samples and their corresponding sequencing data are processed to reconstruct the maximum number of bacterial genomic sequences and predict potential genes. Once this gene collection is complete, the workflow performs advanced annotation, assigning each gene to known or novel bacterial species to detail its biological functions.
While this process is computationally intensive, it is vital. The insights derived from this data underpin every initiative at BioLab, forming the foundation for our understanding of the human gut microbiome’s composition and functions. This opens doors to identifying druggable targets and innovative candidate molecules.
To execute computational tasks and workflows, BioLab activates between 50 and 200 Amazon Elastic Compute Cloud (Amazon EC2) instances for each data analysis. This on-demand approach means that instances are spun up only when needed and deactivated when not in use. Utilizing EC2 Spot instances, a typical data analysis can cost as little as a few dollars per sample, completing in approximately four to six hours. Thanks to AWS and Nextflow, we can execute and analyze complete, automated workflows in parallel within just a few hours, significantly saving time and resources. AWS Batch has streamlined the execution of scientific workloads on the cloud at scale.
A standout feature of Nextflow is its ability to resume entire workflows, ensuring that completed jobs are not unnecessarily re-executed. This efficiency allows us to manage workflow interruptions or implement changes with minimal impact on time and cost while maintaining consistency in pipeline results.
Moreover, both AWS Batch and Nextflow inherently support Docker containers, which encapsulate and facilitate the reuse of existing tools and analytical pipelines. This capability further eases the transition from small development projects to large-scale production environments.
Technologies like AWS Batch and Nextflow empower BioLab to concentrate on data analysis rather than infrastructure management or workflow execution. Consequently, BioLab can focus on developing innovative therapeutic solutions for microbiome-associated diseases without the burden of on-premises infrastructure costs and computational limitations. For those interested in understanding more about health management, consider exploring this resource on health. Additionally, for insights into job roles, the prep cook job description can be extremely informative. For shared experiences, you can check out this Reddit thread about onboarding.