Run large-scale simulations with AWS Batch multi-container jobs

Various industries such as automotive, robotics, and finance are increasingly incorporating computational workloads like simulations, machine learning (ML) model training, and big data analytics to enhance their products. For instance, automakers use simulations to test autonomous driving features, robotics companies utilize ML algorithms to improve robot perception, and financial firms conduct detailed analyses to manage risk, process transactions, and identify fraud. Some of these workloads, particularly simulations, are complex to run due to their diverse components and high computational demands. For example, a driving simulation involves creating 3D virtual environments, vehicle sensor data, vehicle dynamics for car control, and more. Similarly, a robotics simulation may involve testing numerous autonomous delivery robots in a large warehouse setting. AWS Batch is a fully managed service that supports running batch workloads across various AWS compute options like Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), AWS Fargate, and Amazon EC2 Spot or On-Demand Instances. Initially, AWS Batch only allowed single-container jobs and required additional steps to combine all components into a single container. It also did not support using separate “sidecar” containers that provide additional services like data logging. This necessitated coordination among multiple teams for any code changes as rebuilding the entire container was necessary. However, AWS Batch now offers multi-container jobs, simplifying and expediting the execution of large-scale simulations in fields like autonomous vehicles and robotics. These workloads typically involve the simulation itself and the system under test, developed and optimized by different teams. With the ability to run multiple containers per job, AWS Batch provides advanced scaling, scheduling, and cost optimization. Customers like IPG Automotive, MORAI, and Robotec.ai are already leveraging AWS Batch multi-container jobs to run their simulation software in the cloud. Let’s explore how this works practically using a simplified example of a maze-solving simulation running on containers.



Source link