NORDICS20 - Assets

Deep Learning on AWS

Amazon Web Services Resources EMEA

Issue link:

Contents of this Issue


Page 17 of 50

Amazon Web Services Deep Learning on AWS Page 13 involves tasks such as ingesting the data, performing extract, transform, load (ETL), visualizing data, and wrangling data to develop high-quality training dataset for training deep learning models. Amazon Simple Storage Service (Amazon S3) can be used as central storage layer to store and democratize data for deep learning. Your applications can easily achieve thousands of transactions per second by using Amazon S3 as the storage tier for deep learning training jobs. Amazon S3 automatically scales to high request rates. Make sure to consider the throughput between Amazon EC2 and Amazon S3 during ingestion and reading of objects from Amazon S3. You can achieve higher performance using multiple Amazon EC2 instances in a distributed manner. Amazon SageMaker uses Amazon S3 as a storage tier for data used in training jobs and batch inference, and for storing trained models. Amazon SageMaker supports both batch and pipe mode to read data from Amazon S3 in the local Amazon Elastic Block Store (Amazon EBS) volume of Amazon SageMaker training instances. DIY customers who want to manage their own compute clusters on Amazon EC2 can use Amazon S3 as the storage layer or they can use Amazon FSx for Lustre hydrated from Amazon S3 with lazy loading to build a data caching layer for deep learning jobs. Both of the options are available for a DIY setup. You must make a tradeoff between price and performance. Amazon FSx for Lustre Amazon FSx for Lustre is built on open-source Lustre. Lustre is an open-source highly scalable, highly distributed, and highly parallel file system that can be used as a deep learning data caching layer for distributed training. The high-performance capabilities and open licensing make Lustre a popular choice for deep learning workloads. Lustre file systems are scalable and can be used with multiple compute clusters with tens of thousands of client notes, PBs of data, and TB per second of aggregate I/O throughput. If you are training a deep neural network, Lustre provides you with the capability to get the source data fast with low latency. But, setting up a Lustre cluster can be challenging. Amazon FSx for Lustre simplifies the complexity of setting up and managing the Lustre File System and provides an experience that allows you to create a file system in minutes, mount it on any number of clients, and start accessing it right away. Amazon

Articles in this issue

Links on this page

view archives of NORDICS20 - Assets - Deep Learning on AWS