There are multiple cloud computing services available. However, we have chosen to use Amazon EC2 in this tutorial because of its popularity and the free tier use provided. With the free tier, you are able to try the service without incurring any expenses. However you need to be aware of the services provided within the free tier to avoid billing surprises. Amazon provides all that information. Before you can start using the free tier, you need to register and provide a valid credit card.
The service that is specifically used with docker is referred to as Amazon EC2 Container service (ECS). ECS provides a scalable and fast way of starting, stopping and managing containers within Amazon EC2. With ECS, you are able to access common EC2 services, start and stop containerized applications via API calls, and know the cluster state from a single point. With ECS, you are able to control how containers are placed, and create isolation and availability rules. You can use ECS directly or use AWS Elastic Beanstalk. However ECS gives you better control and in this tutorial our discussion will be limited to the use of ECS.
ECS gives you features available within other EC2 services like Elastic load balancing, EBS volumes and security groups. There is really no new skillset required to work with ECS besides what is required in EC2. ECS simply relies on EC2 instances and EBS volumes. ECS has several advantages that we will highlight.
ECS eliminates the need for complex cluster management activities. Developer effort is directed towards creating Docker applications
ECS is a very flexible approach for placing containers. You can use it for applications that require a lot of time to run.
ECS works with other Amazon services such as load balancing and block storage
ECS provides APIs for integrating or extending the service.
With ECS you are able to scale to use of thousands of containers without any additional complexities
The instances you launch on EC2 are not shared with any other person so there is a high level of security
In this tutorial, we will assume registering on Amazon is straight forward so we will not cover the details. Setting up a cluster on Amazon requires several steps. Here we will give a summary but there is a tutorial on setting up a Hadoop cluster that delves into the details, so the reader can refer to that tutorial.
In a nutshell, to set up a cluster you need to login into the EC2 console here https://aws.amazon.com/console/. After logging in, you need to create virtual servers. Here we are only interested in using servers under the free tier so you can apply the free tier filter to show only images under the free tier.
Select a micro instance and the number of instances to spin. For this tutorial, we can select two instances. Once you are confident, you can spin up more instances. Specify the type of storage you would like to use and give an informative description of your cluster.
You need to configure a security group. Because this is just a demonstration tutorial, we will allow connections from any IP address. In a production environment, this would be a security vulnerability. In a production environment it is important to specify IP addresses from which connections are allowed.
The key thing to note here is that you can only download the key pair of an instance only once. Download the key pair and store it in a secure location. After configuring the security group you can then launch your cluster.
To connect to the cluster we have created, we will use SSH. SSH is the protocol that gives us access to a remote server. SSH requires some configuration effort. The setting up Hadoop tutorial demonstrated how to configure SSH so refer to that article. Briefly configuring SSH involves installing openssh and configuring its settings.
Once your cluster has been launched, you are ready to start preparing your containers so that they can be run in the cloud. We begin by defining tasks and services showing the containers and services that will be run in your instances. Just like on non cloud deployments storage of containers happens on Docker registries which may be internal or external to AWS. We will discuss in detail the set up activities that are needed for containers to be run in the cloud.
In earlier Docker containers we discussed containers, images and registries. The concepts remain the same within EC2.
For Dockerized applications to run on ECS we need to prepare a task definition. A task definition is a JSON document that describes containers forming your application. Within the task definition you specify containers to be used, their location, ports and data volumes that will be used. The developer guide provided by Amazon is available here: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_defintions.html and it explains all the parameters that can be specified in a task definition.
After preparing your task definition, you instantiate it as a task and specify how many tasks you will be running in your instance. A task scheduler tool is available on ECS to assist in placement of tasks on container instances. A container agent provides information on tasks that are running, how resources are being used, starting and stopping tasks. Amazon EC2 container registry is the Docker registry service that is managed by Amazon.
You can work with ECS in any of the following ways.
AWS management console- this is a browser interface for resource management
AWS command line- this is an alternative to the management console. It is available as AWS command line interface or AWS tools for Windows Power Shell
Amazon ECS CLI- this enables management of clusters and tasks locally via Docker compose.
AWS SDK- the SDKs enable use of different programming languages.
In this tutorial, we introduced the Amazon cloud computing platform and noted it is popular and easy to use. We discussed ECS which is Amazon’s approach to run containers in the cloud. We identified the strengths of ECS in running containers in the cloud. We discussed how to launch an instance and configure SSH so that you can remotely access an instance. We discussed preparations that need to be done before a container can be run in the cloud. Finally, we discussed ways in which you can use ECS.