Data preprocessing – The CPU generally handles any data preprocessing such as conversion or resizing.Amazon Elastic Block Store (Amazon EBS) volumes aren’t local resources, and involve network operations.

Local disk refers to an instance store, where storage is located on disks that are physically attached to the host computer.

Disk I/O – Read data from local disk into CPU memory.Network operations – Download the data from Amazon Simple Storage Service (Amazon S3).The general steps usually involved in getting the data into the GPU memory are the following: The following diagram illustrates the architecture of optimizing I/O. The challenge is to optimize I/O or the network operations in such a way that the GPU never has to wait for data to perform its computations. The faster you load data into GPU, the quicker it can perform its operation.

For GPUs to perform these operations, the data must be available in the GPU memory. The basicsĪ single GPU can perform tera floating point operations per second (TFLOPS), which allows them to perform operations 10–1,000 times faster than CPUs. You can typically see performance improvements up to 10-fold in overall GPU training by just optimizing I/O processing routines. In this post, we focus on general techniques for improving I/O to optimize GPU performance when training on Amazon SageMaker, regardless of the underlying infrastructure or deep learning framework.

Optimizing communication between GPUs during multi-GPU or distributed trainingĪmazon SageMaker is a fully managed service that enables developers and data scientists to quickly and easily build, train, and deploy machine learning (ML) models at any scale.

Optimizing I/O and network operations to make sure that the data is fed to the GPU at the rate that matches its computations.

Using the latest high performant libraries and GPU drivers.

Optimizing code to make sure that underlying hardware is fully utilized.

However, to fully benefit from the use of GPUs, you should consider the following aspects: GPUs can significantly speed up deep learning training, and have the potential to reduce training time from weeks to just hours.

Image tuner file reader