{"id":14431,"date":"2022-05-11T00:23:34","date_gmt":"2022-05-11T04:23:34","guid":{"rendered":"https:\/\/ptp.cloud\/?p=14431"},"modified":"2025-08-12T03:45:23","modified_gmt":"2025-08-12T07:45:23","slug":"apt-reduces-pipeline-runtime-with-aws-batch-nextflow","status":"publish","type":"post","link":"https:\/\/ptp.cloud\/apt-reduces-pipeline-runtime-with-aws-batch-nextflow\/","title":{"rendered":"How Adaptive Phage Therapeutics Streamlined Data Pipelines with AWS Batch and Nextflow"},"content":{"rendered":"[et_pb_section fb_built=”1″ custom_padding_last_edited=”on|tablet” admin_label=”Section” _builder_version=”4.27.4″ _module_preset=”default” background_color=”#3e489d” background_image=”https:\/\/ptp.cloud\/wp-content\/uploads\/2024\/10\/Square-Pattern-Hero-Background.png” custom_padding=”5px||||false|false” custom_padding_tablet=”40px||40px||true|false” custom_padding_phone=”40px||40px||true|false” da_disable_devices=”off|off|off” locked=”off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row column_structure=”2_3,1_3″ make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_padding=”63px||1px|||” global_colors_info=”{}”][et_pb_column type=”2_3″ module_class=”col-vert-cent” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_heading title=”How Adaptive Phage Therapeutics Reduced Pipeline Runtime by 80% with AWS Batch and Nextflow*” _builder_version=”4.27.4″ _module_preset=”default” title_font=”–et_global_heading_font|700|||||||” title_text_color=”#ffffff” title_font_size=”3.5rem” title_line_height=”1.2em” max_width_tablet=”620px” max_width_phone=”620px” max_width_last_edited=”on|tablet” custom_padding=”40px||||false|false” global_colors_info=”{}”][\/et_pb_heading][\/et_pb_column][et_pb_column type=”1_3″ module_class=”col-vert-cent” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_image src=”https:\/\/ptp.cloud\/wp-content\/uploads\/2024\/10\/PTP-Hero-Section-Goat-Illustration-min.png” alt=”Illustration of Goat working on servers leading data to the cloud and to a proved treatment” title_text=”PTP Hero Section Goat Illustration” align=”center” disabled_on=”on|off|off” _builder_version=”4.27.4″ _module_preset=”default” max_width_tablet=”75%” max_width_phone=”75%” max_width_last_edited=”on|tablet” custom_padding=”25px||25px||true|false” global_colors_info=”{}”][\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_margin=”|auto|29px|auto||” global_colors_info=”{}”][et_pb_column type=”4_4″ module_class=”col-vert-cent” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”#ffffff” text_font_size=”1.5rem” text_line_height=”2.1rem” max_width_tablet=”620px” max_width_phone=”620px” max_width_last_edited=”on|tablet” global_colors_info=”{}”]\n

Adaptive Phage Therapeutics (APT) is a clinical-stage biotech company advancing therapies to combat multi-drug-resistant infections. Traditional antimicrobials are becoming obsolete as pathogens evolve resistance. APT\u2019s innovative PhageBank therapy leverages a growing library of phages, offering evergreen broad-spectrum and polymicrobial coverage. Through a proprietary phage susceptibility assay, commercialized with Mayo Clinic Laboratories, APT is positioned to revolutionize antimicrobial treatment on a global scale. This groundbreaking work is partially funded by the U.S. Department of Defense.<\/span><\/p>\n[\/et_pb_text][dsm_icon_list icon_font_size_tablet=”” icon_font_size_phone=”20px” icon_font_size_last_edited=”on|phone” module_class=”hero-stat” _builder_version=”4.27.4″ _module_preset=”default” text_font_size_tablet=”” text_font_size_phone=”13px” text_font_size_last_edited=”on|phone” locked=”off” global_colors_info=”{}”][dsm_icon_list_child text=”*Decreased from 5-6 hours to just 1.15 hours, an approximate 80% improvement.” font_icon=”||fa||900″ icon_padding=”1px” icon_font_size=”2rem” url_new_window=”on” _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|500|on||||||” text_text_color=”#0ca8fc” text_font_size=”1.2rem” text_line_height=”1.25rem” text_font_size_tablet=”1.2rem” text_font_size_phone=”1rem” text_font_size_last_edited=”on|phone” global_colors_info=”{}”][\/dsm_icon_list_child][\/dsm_icon_list][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ custom_padding=”32px||1px|||” da_disable_devices=”off|off|off” locked=”off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row column_structure=”1_2,1_2″ make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_padding=”8px||1px|||” global_colors_info=”{}”][et_pb_column type=”1_2″ _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font||||||||” text_text_color=”gcid-secondary-color” text_font_size=”28px” text_line_height=”2rem” custom_margin=”45px||||false|false” custom_padding=”||||false|false” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22%93}”]\n

The Challenge<\/strong><\/span><\/h2>\n[\/et_pb_text][\/et_pb_column][et_pb_column type=”1_2″ _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_image src=”https:\/\/ptp.cloud\/wp-content\/uploads\/2025\/01\/adaptive-phage-therapeutics-logo.png” alt=”Logo of Adaptive Phage Therapeutics featuring abstract blue phage shapes and company name in stylish typography.” title_text=”adaptive-phage-therapeutics-logo” url=”https:\/\/www.linkedin.com\/company\/adaptive-phage-therapeutics” url_new_window=”on” align=”right” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_padding=”19px||63px|||” locked=”off” global_colors_info=”{}”][et_pb_column type=”4_4″ module_class=”col-vert-cent” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”gcid-secondary-color” text_font_size=”28px” text_line_height=”2.3rem” custom_margin=”||7px|||” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22,%22text_text_color%22,%22text_text_color%22%93}”]\n

APT<\/a> had an existing pipeline built with Snakemake, a bioinformatics workflow engine. While the team\u2019s cloud environment was effective, they sought to standardize to a High-Performance Compute (HPC) platform that could:<\/span><\/p>\n

\u2022\u00a0 Accelerate pipeline runtimes to deliver refined data faster.<\/span><\/p>\n

\u2022\u00a0 Enable scientists to process data efficiently, reducing pipeline execution times from days to minutes where possible.<\/span><\/p>\n

\u2022\u00a0 Scale dynamically to meet variable computational demands while managing costs.<\/span><\/p>\n

Pipeline jobs, originating from wet bench labs, required scientists to submit raw data through the pipeline. The speed of processing directly impacted the pace of research, making optimization critical.<\/span><\/p>\n[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ custom_padding_last_edited=”on|tablet” next_background_color=”#ffffff” admin_label=”Section” _builder_version=”4.27.4″ _module_preset=”default” background_color=”#3e489d” background_image=”https:\/\/ptp.cloud\/wp-content\/uploads\/2024\/10\/Square-Pattern-Hero-Background.png” custom_padding=”||52px||false|false” custom_padding_tablet=”40px||40px||true|false” custom_padding_phone=”40px||40px||true|false” bottom_divider_style=”arrow” bottom_divider_height=”83px” da_disable_devices=”off|off|off” locked=”off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_padding=”||99px|||” global_colors_info=”{}”][et_pb_column type=”4_4″ module_class=”col-vert-cent” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”gcid-secondary-color” text_font_size=”28px” text_line_height=”2.3rem” custom_margin=”||0px|||” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22,%22text_text_color%22%93}”]\n

The Solution<\/strong><\/p>\n

PTP\u2019s CloudOps Engineering team<\/a> designed a modernized architecture to update APT\u2019s pipeline discovery process, leveraging AWS Batch and migrating from Snakemake to <\/span>Nextflow<\/a>:<\/span><\/p>\n

AWS Batch and Nextflow Integration
<\/span><\/strong>\u2022\u00a0 Pipelines now begin with a job submitted to AWS Batch, which launches an EC2 instance running Nextflow.<\/span><\/p>\n

\u2022\u00a0 Docker images handle pipeline execution, ensuring standardization and repeatability.<\/span><\/p>\n

\u2022\u00a0 Nextflow dynamically provisions instances for parallel processing, improving efficiency and reducing costs.<\/span><\/p>\n

Automated Workflow with Lambda
<\/span><\/strong>\u2022\u00a0 Lambda functions monitor S3 buckets for new data uploads from wet labs and automatically trigger pipeline execution.<\/span><\/p>\n

\u2022\u00a0 Outputs from the pipeline are saved to S3, triggering additional Lambda functions to initiate subsequent pipelines.<\/span><\/p>\n

Optimized Data Storage and Organization
<\/span><\/strong>\u2022\u00a0 All pipeline data is stored and organized in S3, with outputs arranged for downstream analysis.<\/span><\/p>\n

\u2022\u00a0 Polished data is processed through additional pipelines as needed, with results consolidated and presented to scientists.<\/span>\u00a0<\/span><\/p>\n[\/et_pb_text][et_pb_image src=”https:\/\/ptp.cloud\/wp-content\/uploads\/2024\/05\/aws_infrastructure_design_nextflow_workflow.jpg” alt=”A diagram illustrating a Nextflow workflow using AWS infrastructure, including workflow-queue, job-queue, Nextflow head-node, Nextflow processes, and S3 buckets for workflow scripts and input\/output files.” title_text=”aws_infrastructure_design_nextflow_workflow” align=”center” _builder_version=”4.27.4″ _module_preset=”default” custom_margin=”||144px|||” custom_padding=”137px|||||” box_shadow_style=”preset1″ box_shadow_blur=”22px” box_shadow_spread=”27px” box_shadow_color=”#FFFFFF” global_colors_info=”{}”][\/et_pb_image][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”gcid-secondary-color” text_font_size=”28px” text_line_height=”2.3rem” custom_margin=”||0px|||” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22,%22text_text_color%22%93}”]\n

AWS Services Implemented:<\/span><\/strong><\/span><\/p>\n

<\/strong><\/p>\n

<\/strong><\/span><\/strong>Compute and Storage
<\/strong>EC2<\/a>, <\/span>AWS Batch<\/a>, <\/span>Lambda<\/a>, <\/span>ECR<\/a>, and <\/span>S3<\/a><\/p>\n

Governance and Monitoring
<\/strong>CodeCommit<\/a>, CloudFormation<\/a>, CloudWatch<\/a>, CloudTrail<\/a>, IAM<\/a>, and AWS Config<\/a><\/span><\/p>\n

Networking
<\/strong>Virtual Private Cloud (VPC)<\/a><\/span><\/p>\n[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ custom_padding=”32px||1px|||” da_disable_devices=”off|off|off” locked=”off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row make_equal=”on” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_margin=”36px|auto||auto||” custom_padding=”2px||42px|||” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”gcid-secondary-color” text_font_size=”28px” text_line_height=”2rem” custom_margin=”||3px|||” custom_padding=”||11px|||” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22%93}”]\n

The Outcome<\/strong><\/span><\/strong><\/span><\/h2>\n

<\/strong><\/span><\/p>\n

The implementation of AWS Batch and Nextflow delivered transformative results for APT:<\/span><\/p>\n

<\/span><\/span><\/p>\n[\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”–et_global_heading_font|300|||||||” text_text_color=”rgba(0,0,0,0.66)” text_font_size=”28px” text_line_height=”2.3rem” custom_margin=”||41px|||” custom_padding=”33px|25px|33px|25px|true|true” text_font_size_tablet=”28px” text_font_size_phone=”24px” text_font_size_last_edited=”on|phone” border_radii=”on|2px|2px|2px|2px” border_width_all=”1px” global_colors_info=”{%22gcid-secondary-color%22:%91%22text_text_color%22,%22text_text_color%22,%22text_text_color%22,%22text_text_color%22%93}”]\n

Runtime Reduction
<\/strong>Pipeline runtimes decreased from 5-6 hours to just 1.15 hours<\/strong><\/em> <\/span>using approximately 650 CPUs.<\/span><\/p>\n

Parallel Processing
<\/strong>Leveraging AWS Batch allowed pipelines to run concurrently, delivering actionable data faster without additional costs.<\/span><\/p>\n

Enhanced Scalability
<\/strong>The architecture enables APT to handle increasing data volumes efficiently.<\/span><\/p>\n

Extended Team Support
<\/strong>PTP\u2019s CloudOps Engineering team provides ongoing design, architecture, and cloud management support, acting as an extension of APT\u2019s data science team.<\/span><\/p>\n[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row column_structure=”1_2,1_2″ module_class=”vert-cent” _builder_version=”4.27.4″ _module_preset=”default” width=”85%” max_width=”1380px” custom_margin=”0px||0px||false|false” custom_padding=”0px||0px||false|false” locked=”off” global_colors_info=”{}”][et_pb_column type=”1_2″ module_id=”contact” _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_image src=”https:\/\/ptp.cloud\/wp-content\/uploads\/2024\/12\/Graphs-Isometric-Contained-Icon.png” alt=”Graphs Isometric Contained Icon” title_text=”Graphs Isometric Contained Icon” show_bottom_space=”off” disabled_on=”on|off|off” _builder_version=”4.27.4″ _module_preset=”default” width_tablet=”200px” width_phone=”150px” width_last_edited=”on|phone” custom_margin=”|||-20px|false|false” global_colors_info=”{}”][\/et_pb_image][et_pb_heading title=”Ready to optimize your HPC environment? ” _builder_version=”4.27.4″ _module_preset=”default” title_level=”h2″ title_font=”–et_global_heading_font|600|||||||” title_text_align=”left” title_text_color=”gcid-0e52a8f4-5dfd-4102-9f40-fd354ef9edd5″ title_font_size=”2.88rem” title_line_height=”1.2em” max_width_tablet=”100%” max_width_phone=”300px” max_width_last_edited=”on|phone” custom_margin=”15px||5px||false|false” locked=”off” global_colors_info=”{%22gcid-0e52a8f4-5dfd-4102-9f40-fd354ef9edd5%22:%91%22title_text_color%22%93}”][\/et_pb_heading][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”Ubuntu||||||||” text_text_color=”#000000″ text_font_size=”1.25rem” text_line_height=”1.75rem” max_width=”620px” locked=”off” global_colors_info=”{}”]\n

APT\u2019s transition to an AWS Batch and Nextflow-based pipeline architecture underscores the transformative potential of cloud-native solutions. By reducing runtime, scaling dynamically, and enhancing efficiency, APT is better positioned to advance its mission of combating antimicrobial resistance.<\/span><\/p>\n

 <\/p>\n[\/et_pb_text][et_pb_button button_url=”https:\/\/outlook.office365.com\/owa\/calendar\/PTP1@pinnacletechpartners.com\/bookings\/” url_new_window=”on” button_text=”Schedule a call” button_alignment=”left” _builder_version=”4.27.4″ _module_preset=”default” custom_button=”on” button_text_size=”18px” button_text_color=”#ffffff” button_bg_color=”gcid-primary-color” button_border_width=”0px” button_border_radius=”50px” button_font=”Ubuntu|500|||||||” button_use_icon=”off” custom_padding=”0.8rem|1.8rem|0.8rem|1.8rem|true|true” button_text_size_tablet=”1rem” button_text_size_phone=”1rem” button_text_size_last_edited=”on|tablet” locked=”off” global_colors_info=”{%22gcid-primary-color%22:%91%22button_bg_color%22%93,%22gcid-4a2771a4-2bac-479e-b2cf-583957402471%22:%91%22button_bg_color__hover%22%93}” button_bg_color__hover=”#0c71c3″ button_bg_color__hover_enabled=”on|desktop”][\/et_pb_button][\/et_pb_column][et_pb_column type=”1_2″ _builder_version=”4.27.4″ _module_preset=”default” background_color=”#2f348d” custom_padding=”60px|60px|25px|60px|false|true” custom_padding_tablet=”60px|60px|25px|60px|false|true” custom_padding_phone=”30px|30px|0px|30px|false|true” custom_padding_last_edited=”on|tablet” border_radii=”on|10px|10px|10px|10px” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”Ubuntu||||||||” text_text_color=”#FFFFFF” text_font_size=”18px” text_line_height=”1.4em” header_2_font=”Ubuntu|500|||||||” header_2_text_color=”#FFFFFF” header_2_font_size=”38px” header_2_line_height=”1.2em” global_colors_info=”{}”]\n

Let us help you unlock your potential.<\/strong><\/h2>\n

Contact PTP today<\/strong><\/span> to learn how we can help accelerate your research and innovation.<\/span><\/p>\n[\/et_pb_text][et_pb_code _builder_version=”4.27.4″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]

\n
\n
\n
\nHomepage Contact Us<\/legend>\r\n\r\n
\n\n\n\n\n\n
\r\n\t