Deploying Elasticsearch on AWS ECS with Terraform and Ansible: A DevOps Guide

Elasticsearch is a powerful search and analytics engine used for log analysis, full-text search, and real-time data processing. Deploying it on AWS Elastic Container Service (ECS) with infrastructure-as-code tools like Terraform and Ansible ensures scalability, automation, and repeatability. In this guide, we’ll walk you through setting up an Elasticsearch cluster on AWS ECS using EC2 instances, Terraform for infrastructure, and Ansible for configuration management. This blog is perfect for DevOps engineers, AWS practitioners, and developers looking to streamline their Elasticsearch deployments.

What is AWS ECS and Why Choose EC2 Over Fargate? 

Understanding AWS ECS

Amazon Elastic Container Service (ECS) is a fully managed container orchestration platform that simplifies running Docker containers at scale. It integrates seamlessly with other AWS services like Elastic Container Registry (ECR) and CloudWatch, making it ideal for workloads like Elasticsearch.

ECS vs. Fargate: Why EC2?

AWS ECS offers two launch types: EC2 and Fargate. While Fargate is serverless and requires minimal management, EC2 provides greater control over compute resources, which is critical for Elasticsearch’s compute-intensive and storage-heavy workloads. Here’s why EC2 is preferred for this setup:

• Cost Efficiency: EC2 instances, especially with reserved or spot pricing, are often more cost-effective for persistent workloads compared to Fargate’s pay-per-use model.

• Control: EC2 allows customization of instance types, storage (e.g., EBS volumes), and networking, which Elasticsearch benefits from.

• Storage Needs: Elasticsearch requires persistent storage, which is easier to manage with EC2’s EBS volumes than Fargate’s ephemeral storage.

For Elasticsearch, EC2 strikes the right balance between control, cost, and performance, making it the go-to choice for this deployment.

ECS Base Components and Architecture Scheme

Core ECS Components

To deploy Elasticsearch on AWS ECS, you need to understand its core components:

• ECS Cluster: A logical grouping of EC2 instances or Fargate tasks that run containers.

• Task Definitions: JSON configurations defining container images, CPU, memory, and ports.

• Services: Ensure the desired number of tasks are running and integrated with load balancers.

• Container Instances: EC2 instances running the ECS agent to execute tasks.

Architecture Overview

The architecture for this setup includes:

• VPC and Subnets: A Virtual Private Cloud (VPC) with public and private subnets for security and isolation.

• EC2 Instances: Host the ECS cluster, configured with EBS volumes for Elasticsearch data persistence.

• Application Load Balancer (ALB): Routes traffic to Elasticsearch containers.

• Security Groups: Control access to ECS instances and Elasticsearch ports (e.g., 9200).

• EBS Volumes: Provide persistent storage for Elasticsearch data.

This architecture ensures high availability, scalability, and secure access to your Elasticsearch cluster.

Elasticsearch Dockerfile and AWS ECR

Building an Elasticsearch Dockerfile

To run Elasticsearch on ECS, you need a Docker image tailored to your needs. Below is an example Dockerfile using the official Elasticsearch image with custom configurations:

dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:8.10.0
ENV discovery.type=single-node
ENV xpack.security.enabled=false
RUN bin/elasticsearch-plugin install analysis-icu
COPY jvm.options /usr/share/elasticsearch/config/jvm.options.d/

This Dockerfile:

• Uses the official Elasticsearch 8.10.0 image.

• Configures a single-node cluster for simplicity.

• Disables security features (optional, depending on your setup).

• Installs the ICU analysis plugin for enhanced text processing.

• Customizes JVM options for memory settings.

Pushing to AWS ECR

AWS Elastic Container Registry (ECR) stores your Docker images securely. To push the Elasticsearch image:

1. Create an ECR repository: aws ecr create-repository –repository-name elasticsearch.

2. Build the Docker image: docker build -t elasticsearch ..

3. Tag the image: docker tag elasticsearch:latest <account-id>.dkr.ecr.<region>.amazonaws.com/elasticsearch:latest.

4. Push to ECR: docker push <account-id>.dkr.ecr.<region>.amazonaws.com/elasticsearch:latest.

An Ansible playbook can automate this process, ensuring consistency across environments.

ECS IAM Profile Terraform Module

Why IAM Roles Matter

IAM roles ensure secure communication between ECS tasks, EC2 instances, and AWS services like ECR and CloudWatch. You need two roles:

• Task Execution Role: Allows ECS tasks to pull images from ECR and send logs to CloudWatch.

• EC2 Instance Profile: Enables EC2 instances to register with the ECS cluster.

Terraform Module for IAM

Below is a sample Terraform module for IAM roles:

hcl 

module "ecs_iam" {
  source = "./modules/ecs-iam"

  role_name = "ecs-task-execution-role"
  instance_profile_name = "ecs-instance-profile"
}

# modules/ecs-iam/main.tf
resource "aws_iam_role" "ecs_task_execution" {
  name = var.role_name
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_policy" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_instance_profile" "ecs_instance" {
  name = var.instance_profile_name
  role = aws_iam_role.ecs_task_execution.name
}

This module creates reusable, secure IAM roles for your ECS deployment.

ECS EC2 Terraform Module

Provisioning EC2 Instances

EC2 instances serve as container hosts in the ECS cluster. A Terraform module can automate their setup, including Auto Scaling for resilience.

Terraform Module for EC2

Here’s a sample module:

hcl 

module "ecs_ec2" {
  source = "./modules/ecs-ec2"

  instance_type = "t3.medium"
  min_size      = 2
  max_size      = 4
  vpc_id        = aws_vpc.main.id
  subnet_ids    = aws_subnet.private[*].id
}

# modules/ecs-ec2/main.tf
resource "aws_launch_template" "ecs_ec2" {
  name_prefix   = "ecs-ec2-"
  image_id      = "ami-0c55b159cbfafe1f0" # ECS-optimized AMI
  instance_type = var.instance_type
  user_data     = base64encode("#!/bin/bash\necho ECS_CLUSTER=${var.cluster_name} >> /etc/ecs/ecs.config")
}

resource "aws_autoscaling_group" "ecs_asg" {
  min_size         = var.min_size
  max_size         = var.max_size
  vpc_zone_identifier = var.subnet_ids
  launch_template {
    id = aws_launch_template.ecs_ec2.id
  }
}

This module provisions EC2 instances with an ECS-optimized AMI and Auto Scaling for high availability.

ECS Cluster Terraform Module

Setting Up the ECS Cluster

The ECS cluster hosts your Elasticsearch containers. A Terraform module defines the cluster, task definitions, and services.

Terraform Module for ECS Cluster

Here’s a sample module:

hcl 

module "ecs_cluster" {
  source = "./modules/ecs-cluster"

  cluster_name    = "elasticsearch-cluster"
  task_cpu        = 1024
  task_memory     = 2048
  container_image = "<account-id>.dkr.ecr.<region>.amazonaws.com/elasticsearch:latest"
}

# modules/ecs-cluster/main.tf
resource "aws_ecs_cluster" "main" {
  name = var.cluster_name
}

resource "aws_ecs_task_definition" "elasticsearch" {
  family                   = "elasticsearch"
  network_mode             = "awsvpc"
  requires_compatibilities = ["EC2"]
  cpu                      = var.task_cpu
  memory                   = var.task_memory
  container_definitions = jsonencode([{
    name  = "elasticsearch"
    image = var.container_image
    essential = true
    portMappings = [{ containerPort = 9200, hostPort = 9200 }]
  }])
}

resource "aws_ecs_service" "elasticsearch" {
  name            = "elasticsearch-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.elasticsearch.arn
  desired_count   = 2
  launch_type     = "EC2"
  network_configuration {
    subnets         = var.subnet_ids
    security_groups = var.security_group_ids
  }
}

This module sets up a scalable ECS cluster for Elasticsearch.

Deploying an ECS Cluster Using AWS ECS

Step-by-Step Deployment

1. Initialize Terraform: Run terraform init and terraform apply to provision IAM roles, EC2 instances, and the ECS cluster.

2. Use Ansible: Configure EC2 instances and push the Elasticsearch image to ECR using an Ansible playbook.

3. Set Up ALB: Create an Application Load Balancer to route traffic to port 9200.

4. Verify Deployment: Check Elasticsearch cluster health via curl http://<alb-dns>:9200/_cluster/health.

Best Practices

• Use Terraform workspaces for development, staging, and production environments.

• Enable CloudWatch logging for ECS tasks.

• Monitor Elasticsearch performance using CloudWatch or tools like Kibana.

Elasticsearch at AWS ECS – Short Summary

Deploying Elasticsearch on AWS ECS with EC2, Terraform, and Ansible offers a scalable, cost-effective solution. Terraform automates infrastructure setup, while Ansible streamlines configuration and image deployment. This setup leverages EC2’s control and EBS storage for Elasticsearch’s needs, ensuring high availability and performance. For advanced setups, explore multi-node clusters or security features like X-Pack.

Conclusion

This guide demonstrated how to deploy Elasticsearch on AWS ECS using Terraform and Ansible, covering ECS architecture, Dockerfile creation, ECR integration, and Terraform modules for IAM, EC2, and ECS clusters. Try this setup for your next project and explore further AWS DevOps tools to enhance your workflows.

About the Author

Tutor @ Eduonix

Administrator

View All Posts