Amazon Web Services Deep Learning on AWS
Page 37
this section, we review some advanced use cases using Amazon SageMaker and other
AWS services.
Orchestrate Your End-to-End Machine Learning Pipeline using AWS
Step Functions
AWS Step Functions allow you to build resilient serverless workflows. In AWS Step
Functions, a workflow is implemented as a finite state machine. The states can be a
task, a choice, a branch of logic, a set of parallel tasks, an error handler, and so on. The
workflow is implemented as a Directed Acyclic Graph (DAG) and uses GoTo logic. AWS
Step Functions also allows you to throw an exception and do error handling to make the
flow more robust.
In AWS Step Functions, the task states do most of the heavy lifting. There are two types
of task states: Activity task and Lambda task. In Activity tasks, worker requests work
from AWS Step Functions, then takes the work and returns the results. The Lambda
task is a synchronous call to an AWS Lambda function from AWS Step Functions. The
Lambda task has a maximum timeout of 15 minutes as defined by the max execution
duration of the Lambda function. AWS Step Functions also allows you to insert human
actions such as approval and rejection into the state machine. The actions can be used
in the workflow to approve or deny the model push into the production environment.
Using all of the capabilities of AWS Step Functions, you can build a complex end-to-end
deep learning workflow. You can trigger the workflow when the new data arrives in
Amazon S3, start the training job, and deploy the newly trained model. You can make
the workflow more robust and transparent by adding notifications and error handling to
it. The following workflow diagram is a sample representation of an end-to-end deep
learning workflow implemented using AWS Step Functions for retraining and
redeployment.
