NORDICS20 - Assets

Deep Learning on AWS

Amazon Web Services Resources EMEA

Issue link:

Contents of this Issue


Page 18 of 50

Amazon Web Services Deep Learning on AWS Page 14 FSx for Lustre is a fully managed service, so there's nothing to maintain and nothing to administer. You can build standalone file systems for ephemeral use, or you can seamlessly join them to an S3 bucket and then access the contents of the bucket as if it were a Lustre file system. Amazon FSx for Lustre is designed for workloads that require high levels of throughput, IOPS, and consistent low-latencies. One unique feature of Amazon FSx for Lustre is its deep integration with Amazon S3 that allows lazy loading of data into the actual file system. If a customer doesn't know which object to load from the S3 bucket, the Amazon FSx for Lustre loads only the metadata comprised of names, dates, sizes, and so forth for the objects themselves, but it does not load the actual file data until it is required. By default, Amazon S3 objects are only loaded into the file system when first accessed by your applications. If your applications access objects that haven't yet been loaded into your file system, Amazon FSx for Lustre automatically loads the corresponding objects from Amazon S3. Amazon Elastic File System (Amazon EFS) When selecting a storage solution, there is a tradeoff between data locality and a centrally managed storage solution. Amazon EFS is well-suited to support a broad spectrum of use cases—from highly parallelized, scale-out workloads that require the highest possible throughput to single-threaded, latency-sensitive workloads. However, when running batch processing on central locations, Amazon EFS is likely the most suitable storage solution. Amazon EFS enables you to provide easy access to your large machine learning datasets or shared code, right from your notebook environment, without the need to provision storage or worry about managing the network file system yourself. Amazon EFS scales automatically as more data is ingested. Data is stored redundantly across multiple Availability Zones and the performance scales up to 10+ GB per second of throughput as your data grows. Amazon EFS can be simultaneously mounted on thousands of Amazon EC2 instances from multiple Availability Zones. As shown in the diagram below, up to thousands of Amazon EC2 instances from multiple Availability Zones can connect concurrently to a file system. It can also be mounted on multiple Amazon SageMaker Jupyter Notebooks. This feature allows Amazon EFS to be used for data and code sharing, enabling collaboration among deep learning engineers and deep learning scientists. You can also use Amazon EFS as a caching layer for training datasets in distributed training jobs. The following figure shows how you can add an Amazon EFS endpoint to all ephemeral compute nodes to mount a centrally accessible storage solution. Most importantly, this

Articles in this issue

Links on this page

view archives of NORDICS20 - Assets - Deep Learning on AWS