aws cloud devops

Overview

This document provides a comprehensive guide for deploying a Docker application to AWS ECS (Elastic Container Service) and ECR (Elastic Container Registry) using Jenkins. The deployment process involves building a Docker image, pushing it to ECR, updating the ECS task definition, and deploying the updated task definition to an ECS service.

Project Description: Deploying a Simple HTML Web Application on AWS ECS Using CI/CD and Load Testing

Introduction

In this project, we explore the end-to-end process of deploying a simple HTML web application using Amazon Web Services (AWS) Elastic Container Service (ECS). The deployment is facilitated by a Continuous Integration and Continuous Deployment (CI/CD) pipeline, and the application’s performance is validated through load testing. This detailed guide covers everything from setting up the necessary infrastructure to monitoring application performance, offering insights into best practices for modern cloud-native application deployment and management.

Objectives

Set Up Jenkins and CI/CD Pipeline:
- Configure Jenkins on AWS EC2.
- Set up Jenkins plugins for Docker, Amazon ECR, and pipeline management.
- Automate the build, test, and deployment process using Jenkins.
Dockerize the Application:
- Create a Dockerfile for the HTML web application.
- Build and push Docker images to Amazon ECR.
Deploy on AWS ECS:
- Configure the necessary AWS infrastructure (VPC, subnets, security groups).
- Deploy the Dockerized application to ECS with appropriate task definitions and service configurations.
Load Testing and Monitoring:
- Perform load testing using tools like Apache Benchmark (ab) and wrk.
- Monitor application performance using Amazon CloudWatch.
- Set up CloudWatch alarms to alert on performance issues.

Key Steps

Jenkins Setup and CI/CD Pipeline:
- Installation and configuration of Jenkins.
- Setting up Jenkins plugins for a seamless CI/CD process.
- Writing and configuring pipeline scripts for automated deployment.
AWS Infrastructure Configuration:
- Setting up VPC, subnets, and security groups to ensure proper network configuration for ECS tasks.
- Creating and attaching IAM roles with necessary permissions for ECS services.
Dockerization and Deployment:
- Writing a Dockerfile for the HTML web application.
- Building Docker images and pushing them to Amazon ECR.
- Configuring and deploying ECS tasks and services.
Performance Testing and Monitoring:
- Conducting load testing to evaluate application performance under stress.
- Using Amazon CloudWatch for real-time monitoring and setting up alerts for proactive issue resolution.

Outcomes

Automated Deployment:
- Successful setup of a CI/CD pipeline that automates the build and deployment process.
- Reduced manual intervention and errors, speeding up the deployment cycle.
Scalable and Monitored Application:
- The HTML web application is containerized and deployed on a scalable ECS cluster.
- Continuous performance monitoring ensures the application remains reliable and performant under load.
Insights from Load Testing:
- Valuable data on how the application handles increased traffic.
- Identified and resolved potential performance bottlenecks.

Conclusion

This project provides a comprehensive guide for deploying a simple HTML web application on AWS ECS using a CI/CD pipeline. From initial setup to performance testing and monitoring, each step is detailed to ensure a thorough understanding of the deployment process. By following this guide, you will gain hands-on experience in cloud-native application deployment, continuous integration, continuous deployment, and performance monitoring, equipping you with the skills necessary for managing modern cloud-based applications effectively.

GitHub Project Repo: https://github.com/prafulpatel16/ecs-demo.git

Prerequisites

AWS Account: Ensure you have an active AWS account. Docker: Install Docker on your local machine. AWS CLI: Install and configure the AWS CLI. AWS IAM Role: Create an IAM role with permissions for ECS, ECR, and other related services. Jenkins: Install Jenkins on your local machine or use a Jenkins server. Git Repository: Set up a Git repository (e.g., GitHub, GitLab).

Create AWS IAM Role

Create ECS Task Role Name: ecsTaskExecutionRole

Create JSON Policy: EC2ContainerRegistryReadOnly


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:DescribeRepositories",
                "ecr:GetDownloadUrlForLayer",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeImageScanFindings",
                "ecr:ListTagsForResource",
                "ecr:DescribeRegistry",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}

Prerequisites

AWS Account: Ensure you have an active AWS account.
Docker: Install Docker on your local machine.
AWS CLI: Install and configure the AWS CLI.
Code Repository: Set up a Git repository (e.g., GitHub, GitLab).
CI/CD Tool: Use a CI/CD tool like GitHub Actions, GitLab CI, or AWS CodePipeline.

Tools and AWS Services Used for the Project

Tools

Jenkins
- Description: An open-source automation server used to automate the building, testing, and deployment of applications.
- Usage: Set up CI/CD pipelines to automate the build and deployment process.
Docker
- Description: A platform for developing, shipping, and running applications in containers.
- Usage: Containerize the HTML web application.
Apache Benchmark (ab)
- Description: A tool for benchmarking the performance of HTTP web servers.
- Usage: Perform load testing on the deployed application.

AWS Services

Amazon Elastic Container Service (ECS)
- Description: A fully managed container orchestration service.
- Usage: Deploy and manage the containerized HTML web application.
Amazon Elastic Container Registry (ECR)
- Description: A fully managed Docker container registry that makes it easy to store, manage, and deploy Docker container images.
- Usage: Store and manage Docker images for deployment on ECS.
Amazon Elastic Compute Cloud (EC2)
- Description: A web service that provides resizable compute capacity in the cloud.
- Usage: Host Jenkins for setting up the CI/CD pipeline.
Amazon Virtual Private Cloud (VPC)
- Description: A service that lets you launch AWS resources in a logically isolated virtual network.
- Usage: Create a secure and isolated network environment for ECS tasks and services.
Amazon CloudWatch
- Description: A monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers.
- Usage: Monitor application performance, set up alarms, and gain insights into resource utilization.
AWS Identity and Access Management (IAM)
- Description: A web service that helps you securely control access to AWS services and resources.
- Usage: Manage permissions and roles for ECS tasks and services, ensuring secure access control.
AWS Secrets Manager
- Description: A service to help you protect access to your applications, services, and IT resources without the upfront cost and maintenance of hardware security modules (HSMs).
- Usage: Manage and retrieve secrets such as database credentials securely.
Amazon Route 53
- Description: A scalable and highly available Domain Name System (DNS) web service.
- Usage: Route traffic to the application deployed on ECS.
Amazon Simple Notification Service (SNS)
- Description: A fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
- Usage: Send email notifications about the status of the ECS tasks and alarms set in CloudWatch.

Additional Features

Email Notifications Using SNS
- Description: Set up Amazon SNS to send email notifications for important events or alarms related to the ECS deployment.
- Usage: Create an SNS topic, subscribe an email endpoint to the topic, and configure CloudWatch to send notifications to the SNS topic.

Set Up SNS Topic and Email Subscription

Create SNS Topic:

  aws sns create-topic --name ecs-deployment-notifications

Subscribe Email to SNS Topic:

  aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications --protocol email --notification-endpoint your-email@example.com

Configure CloudWatch Alarms to Send Notifications

Create CloudWatch Alarm:

  aws cloudwatch put-metric-alarm --alarm-name "HighCPUUtilization" --metric-name "CPUUtilization" --namespace "AWS/ECS" --statistic "Average" --period 300 --threshold 80 --comparison-operator "GreaterThanThreshold" --dimensions Name=ClusterName,Value=your-cluster-name --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications

By integrating Amazon SNS and email notifications, you ensure that you receive timely updates on the status of your ECS deployments and any critical alerts, thereby enhancing the observability and reliability of your deployment process.

Deployment Steps:

1. Prepare the HTML Web App

2. Dockerize the Application

3. Create a Docker Repository on AWS ECR

4. Push the Docker Image to ECR

5. Create ECS Cluster and Task Definition

6. Create ECS Service

7. Set Up CI/CD Pipeline

8. Monitor with CloudWatch

9. Test and Validate

Implementation in Action

1. Prepare the HTML Web App

2. Dockerize the Application

# Use an official nginx image as the base image
FROM nginx:alpine

# Copy the HTML file to the nginx directory
COPY index.html /usr/share/nginx/html

# Expose port 80
EXPOSE 80

# Start nginx when the container launches
CMD ["nginx", "-g", "daemon off;"]

Build Docker Image: Build docker image in local

docker build -t simple-html-web-app .

docker image build successful

docker images

Let's the docker image locally

Start the docker image 'simple-html-web-app'

docker run -d -p 8080:80 simple-html-web-app:latest

Verify the docker image web app into Browser

Migrate the Docker application to AWS ECS Container platform

3. Create a Docker Repository on AWS ECR

Create ECR Repository:

aws ecr create-repository --repository-name simple-html-web-app

Login to ECR:

aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com

4. Push the Docker Image to ECR

docker tag simple-html-web-app:latest <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest

docker push <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest

5. Create ECS Cluster and Task Definition

Create ECS Cluster:

aws ecs create-cluster --cluster-name simple-html-web-app-cluster

Create Task Definition:

taskdef.json

{
  "family": "simple-html-web-app-task",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "simple-html-web-app-container",
      "image": "<your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ],
      "memory": 512,
      "cpu": 256
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::<your-aws-account-id>:role/ecsTaskExecutionRole"
}

Register Task Definition:

aws ecs register-task-definition --cli-input-json file://task-definition.json

6. Create ECS Service

Create Service:

from terminal using aws cli

aws ecs create-service \
  --cluster simple-html-web-app-cluster \
  --service-name simple-html-web-app-service \
  --task-definition simple-html-web-app-task \
  --desired-count 1 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[<your-subnet-id>],securityGroups=[<your-security-group-id>],assignPublicIp=ENABLED}"

from AWS Console

Networking: It is going to leverage default VPC and it's default security group at the moment

Create new Application Load Balancer

New ECS Service created

Set Up Jenkins CI/CD Pipeline

Jenkins installation

Create EC2 machine and Jenkins Server and install Jenkins

https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/

Part 1: Set Up Jenkins on AWS

Step 1: Launch an EC2 Instance for Jenkins

Sign in to AWS Management Console.
Launch EC2 Instance:
- Go to Services and select EC2.
- Click on Launch Instance.
Choose AMI:
- Select an Amazon Linux 2 AMI (or Ubuntu, if preferred).
Choose Instance Type:
- Select t2.medium (or another type based on your needs).
Configure Instance:
- Configure VPC and Subnet.
- Enable Auto-assign Public IP.
Add Storage:
- Configure storage as needed (8GB or more).
Add Tags:
- Add tags for easy identification (e.g., Key: Name, Value: Jenkins).
Configure Security Group:
- Create a new security group with the following inbound rules:
  - HTTP: Port 80, Source: 0.0.0.0/0
  - Custom TCP Rule: Port 8080, Source: 0.0.0.0/0
  - SSH: Port 22, Source: 0.0.0.0/0
Review and Launch:
- Review your settings and launch the instance.
- Select or create a key pair for SSH access.

Connect to the EC2 Instance:
- Use SSH to connect to your EC2 instance.

    shCopy codessh -i "your-key-pair.pem" ec2-user@your-ec2-public-dns

Install Java:

  shCopy codesudo yum update -y
  sudo amazon-linux-extras install java-openjdk11 -y

Add Jenkins Repository and Install Jenkins:

  shCopy codesudo wget -O /etc/yum.repos.d/jenkins.repo \
      https://pkg.jenkins.io/redhat-stable/jenkins.repo
  sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
  sudo yum install jenkins -y

Start Jenkins:

  shCopy codesudo systemctl start jenkins
  sudo systemctl enable jenkins

Open Jenkins in Browser:
- Navigate to http://your-ec2-public-dns:8080 in your browser.
- Retrieve the initial admin password:

      shCopy codesudo cat /var/lib/jenkins/secrets/initialAdminPassword

Unlock Jenkins:
- Paste the retrieved password to unlock Jenkins.
- Install suggested plugins during setup.

Configure Jenkins Credentials for AWS integration

Global

Add Credentials

Create a jenkins user in AWS IAM role with access key and id and provide those credentials in jenkins

Part 2: Install Required Jenkins Plugins

Install Plugins:
- Go to Manage Jenkins > Manage Plugins > Available.
- Search and install the following plugins:
  - Docker Pipeline
  - Amazon ECR
  - Pipeline
  - Git

Part 3: Configure Jenkins Pipeline for ECS

Step 1: Set Up AWS Credentials in Jenkins

Create AWS IAM User for Jenkins:
- Go to IAM > Users > Add user.
- Username: jenkins-user, Access type: Programmatic access.
- Attach existing policies directly: AmazonEC2ContainerRegistryFullAccess, AmazonECS_FullAccess, AmazonS3FullAccess.
- Download the Access key ID and Secret access key.
Configure Credentials in Jenkins:
- Go to Manage Jenkins > Manage Credentials.
- Under the appropriate domain (e.g., (global)), add a new AWS Credentials:
  - Kind: AWS Credentials
  - Access Key ID and Secret Access Key from the IAM user.

Step 2: Create Jenkins Pipeline

Create a New Pipeline Job:
- Go to Jenkins > New Item.
- Enter a name (e.g., ECS-Deploy-Pipeline), select Pipeline, and click OK.
Configure Pipeline Script:
- In the Pipeline section, choose Pipeline script.
- Enter the following example script:

Create a Pipline CICD

pipeline {
    agent any

    parameters {
        string(name: 'IMAGE_TAG', defaultValue: 'latest', description: 'Docker image tag')
    }

    environment {
        AWS_ACCOUNT_ID = '913151559'
        AWS_REGION = 'us-east-1'
        ECR_REPO = 'simple-html-web-app'
        ECR_REPO_URI = '9313151559.dkr.ecr.us-east-1.amazonaws.com/simple-html-web-app'
        REGISTRY_CREDENTIAL = 'aws-credentials-id' // Ensure this matches the credentials ID in Jenkins
        IMAGE_TAG = "latest-${env.BUILD_NUMBER}" // Unique tag for each build
        ECS_CLUSTER = 'simple-html-web-app-cluster'
        ECS_SERVICE = 'ecs-demo-srv'
        TASK_DEFINITION_FAMILY = 'ecs-task-def'
    }

    stages {
        stage('Test Credentials') {
            steps {
                script {
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh 'aws sts get-caller-identity'
                    }
                }
            }
        }

        stage('Clone Git Repository') {
            steps {
                checkout([$class: 'GitSCM',
                    branches: [[name: '*/master']],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [],
                    submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/prafulpatel16/ecs-demo.git']]
                ])
            }
        }

        stage('Build and Push Docker Image') {
            steps {
                script {
                    // Build and tag Docker image with specified tag
                    sh "docker build -t ${ECR_REPO_URI}:${params.IMAGE_TAG} ."

                    // Login to AWS ECR
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh """
                        aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REPO_URI}
                        """
                    }

                    // Push Docker image to ECR
                    sh "docker push ${ECR_REPO_URI}:${params.IMAGE_TAG}"
                }
            }
        }

        stage('Update ECS Task Definition') {
            steps {
                script {
                    // Load and update task definition JSON
                    def taskDefinition = readFile 'taskdef.json'
                    taskDefinition = taskDefinition.replace("REPLACE_WITH_IMAGE_TAG", "${ECR_REPO_URI}:${params.IMAGE_TAG}")

                    // Write updated task definition to file
                    writeFile file: 'taskdef.json', text: taskDefinition

                    // Register updated task definition and capture the revision number
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        def registerOutput = sh(script: "aws ecs register-task-definition --cli-input-json file://taskdef.json", returnStdout: true).trim()
                        echo "Register Output: ${registerOutput}"
                        def json = readJSON text: registerOutput
                        def taskDefinitionArn = json.taskDefinition.taskDefinitionArn
                        echo "Task Definition ARN: ${taskDefinitionArn}"
                        def taskDefinitionRevision = taskDefinitionArn.tokenize(':').last()  // Extract revision number
                        echo "Task Definition Revision: ${taskDefinitionRevision}"

                        // Save the new task definition revision for later use
                        env.TASK_DEFINITION_REVISION = taskDefinitionRevision
                    }
                }
            }
        }

        stage('Deploy ECS Service') {
            steps {
                script {
                    // Update ECS service with the new task definition revision
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh """
                        aws ecs update-service --cluster ${ECS_CLUSTER} --service ${ECS_SERVICE} --task-definition ${TASK_DEFINITION_FAMILY}:${env.TASK_DEFINITION_REVISION} --force-new-deployment
                        """
                    }
                }
            }
        }
    }
}

CICD Pipeline is ready to build and deploy

Before Deployment verify the ECR repo and ECS task definition

ECR

ECS Task Definition

Let's build the code and deploy

Build Manual

Build and deploy successful

Access Web application

Hit the ALB url: http://ecs-alb-611614342.us-east-1.elb.amazonaws.com/

Verify post deployment that the new task definition has been created

Let's Change and update the code to verify that the new changes successfully apply within the CICD pipeline

Observe that on the web application there is text "Demo ECS" needs to be removed from the code and new code needs to be deployed through CICD

Go to Local VS code and remove the text

After removal the text

Commit the changes and push to the GitHub repo

git push

Expectation: New changes should be updated successfully on the web application

So if we look at the ECS Cluster definition there is 'ecs-task-def:28' which is curently running with the old web application and once the new changes is deployed it should be new 'ecs-task-def:29' should be created with the new changes.

Now let's build the code and push the changes through Jenkins CICD

Build#18

Build Successful

new 'ecs-task-def:29' deploying

deployed

Let's verify the web application has updated the changes successfully

removed text "demo-ecs"

Monitoring CloudWatch

Certainly! Here’s a comprehensive guide to performing a CPU load test on an ECS cluster web app and monitoring CPU utilization on CloudWatch using Apache Benchmark (ab) and wrk from an Amazon Linux 2 AMI:

Step 1: Set Up Amazon Linux 2 Instance

Launch an Amazon Linux 2 Instance:
- Open the EC2 Dashboard in the AWS Management Console.
- Launch a new instance and select Amazon Linux 2 AMI.
- Choose an instance type (e.g., t2.micro).
- Configure instance details, add storage, and configure security groups to allow SSH access.
- Launch the instance and connect to it via SSH.

Step 2: Install Load Testing Tools on Amazon Linux 2

Apache Benchmark (ab)

Install Apache Benchmark:

 sudo yum update -y
 sudo yum install httpd-tools -y

wrk

Install wrk:

 sudo yum install -y git gcc
 git clone https://github.com/wg/wrk.git
 cd wrk
 make
 sudo cp wrk /usr/local/bin

Step 3: Perform Load Testing

Using Apache Benchmark (ab)

Run a Basic Load Test:
```
 ab -n 1000 -c 50 http://your-alb-url/
```
- -n 1000: Number of requests to perform.
- -c 50: Number of multiple requests to perform at a time.
- Replace http://your-alb-url/ with your actual ALB URL.

Increase the Load:

 ab -n 10000 -c 200 http://your-alb-url/

Using wrk

Run a Basic Load Test:
```
 wrk -t12 -c400 -d30s http://your-alb-url/
```
- -t12: Number of threads to use.
- -c400: Number of connections to open.
- -d30s: Duration of the test (30 seconds).
- Replace http://your-alb-url/ with your actual ALB URL.

Step 4: Monitor CPU Utilization on CloudWatch

Enable CloudWatch Monitoring for ECS Cluster

Ensure CloudWatch Monitoring is Enabled:
- Navigate to the ECS Cluster in the AWS Management Console.
- Go to the "Monitoring" tab and ensure CloudWatch metrics are enabled.

Create CloudWatch Alarms

Create an Alarm for CPU Utilization:
- Open the CloudWatch Dashboard in the AWS Management Console.
- Click on "Alarms" > "Create Alarm".
- Select the ECS cluster's CPUUtilization metric.
- Set the threshold (e.g., CPU utilization > 80% for 5 minutes).
- Configure actions (e.g., send an SNS notification).
- Review and create the alarm.

View Metrics

View CPU Utilization Metrics:
- In the CloudWatch Dashboard, navigate to "Metrics".
- Select "ECS" and find the CPU utilization metrics for your cluster and services.

Step 5: Verify and Analyze

Verify Load Test Results:
- Check the output of ab or wrk for request statistics, including requests per second, mean response time, and more.
- Ensure your web application is handling the load as expected.
Analyze CloudWatch Metrics:
- Go to the CloudWatch Dashboard and check the CPU utilization metrics.
- Ensure that the ECS tasks are scaling properly based on the load and the alarms are triggering as expected.

Example CloudWatch Alarm JSON (Optional)

If you prefer to create the CloudWatch alarm using AWS CLI, here is an example JSON configuration:

{
  "AlarmName": "ECS-CPU-Utilization-High",
  "AlarmDescription": "Alarm when ECS CPU utilization exceeds 80%",
  "ActionsEnabled": true,
  "OKActions": [],
  "AlarmActions": [
    "arn:aws:sns:us-east-1:123456789012:MySNSTopic"
  ],
  "MetricName": "CPUUtilization",
  "Namespace": "AWS/ECS",
  "Statistic": "Average",
  "Dimensions": [
    {
      "Name": "ClusterName",
      "Value": "your-ecs-cluster-name"
    },
    {
      "Name": "ServiceName",
      "Value": "your-ecs-service-name"
    }
  ],
  "Period": 300,
  "EvaluationPeriods": 1,
  "Threshold": 80.0,
  "ComparisonOperator": "GreaterThanThreshold"
}

You can use the AWS CLI to create the alarm with the above configuration:

aws cloudwatch put-metric-alarm --cli-input-json file://alarm.json

Replace [file://alarm.json](file://alarm.json) with the path to your JSON file.

By following these steps, you can perform load testing on your ECS cluster web application and monitor its CPU utilization using CloudWatch.

Create Alarm

Select Metric

Increase the Load:

SSH to the Load test machine

ab -n 10000 -c 200 http://your-alb-url/

Hit the ALB URL 10 to 20 times and observe the load

Observer the CPU Load

ALarm Triggered

Email Notification triggered

High CPU Utlilization

Project Conclusion

Overview

The objective of this project was to deploy a simple HTML web application on AWS ECS using a CI/CD pipeline and to ensure the application’s performance under load through load testing and monitoring. This comprehensive exercise demonstrated the full lifecycle of a cloud-native application, from development and deployment to performance monitoring and scaling.

Key Steps and Achievements

Setting Up Jenkins and CI/CD Pipeline:
- Installed and configured Jenkins on an AWS EC2 instance.
- Set up necessary Jenkins plugins for Docker, Amazon ECR, and pipeline management.
- Created a Jenkins pipeline script to build, push Docker images to ECR, and deploy to ECS.
Dockerization and Deployment to AWS ECS:
- Created a Dockerfile for the simple HTML web application.
- Built Docker images and pushed them to Amazon ECR.
- Deployed the application to AWS ECS with appropriate task definitions and service configurations.
Infrastructure Setup on AWS:
- Configured necessary VPC, subnets, and security groups to ensure the ECS tasks had proper networking configurations.
- Created IAM roles and policies to grant necessary permissions for ECS tasks and services.
Load Testing and Monitoring:
- Performed load testing using Apache Benchmark (ab) and wrk to simulate user traffic and evaluate the application's performance.
- Monitored ECS cluster and service performance using Amazon CloudWatch.
- Configured CloudWatch alarms to alert when CPU utilization thresholds were exceeded.
Troubleshooting and Optimization:
- Resolved common issues such as task network configuration errors and IAM role permissions.
- Tuned ECS service and task definitions for optimal performance under load.

Project Outcomes

Successful Deployment:
- The HTML web application was successfully containerized and deployed on AWS ECS.
- The CI/CD pipeline automated the process of building, testing, and deploying the application, ensuring efficient and reliable deployments.
Effective Load Testing:
- Load tests provided valuable insights into the application's performance and scalability.
- Identified and mitigated potential bottlenecks, ensuring the application could handle increased traffic.
Robust Monitoring and Alerts:
- CloudWatch metrics and alarms enabled proactive monitoring of the application's performance.
- Ensured timely alerts and response to any performance degradation or failures.

Lessons Learned

Automation is Key:
- Automating the deployment process using Jenkins and ECS significantly reduces manual errors and speeds up the deployment process.
Importance of Monitoring:
- Continuous monitoring and proactive alerting are crucial for maintaining application performance and availability.
Thorough Testing:
- Load testing is essential to understand how the application performs under various traffic conditions and to ensure it can scale appropriately.

Future Work

Advanced CI/CD Features:
- Implementing advanced CI/CD features such as blue-green deployments or canary releases to minimize downtime and reduce risk during updates.
Enhanced Monitoring:
- Integrating more sophisticated monitoring and logging tools such as AWS X-Ray for distributed tracing and deeper insights into application performance.
Security Enhancements:
- Implementing more robust security measures, including using AWS Secrets Manager for managing sensitive information and enhancing IAM policies.

Conclusion

This project demonstrated the end-to-end process of deploying a web application on AWS ECS, from setting up the CI/CD pipeline to ensuring the application’s performance under load. Through this exercise, key skills in cloud infrastructure management, continuous integration, continuous deployment, and performance monitoring were reinforced, providing a solid foundation for managing and scaling cloud-native applications effectively.

Key Takeaways from the Project

Hands-On Experience with AWS Services:
- ECS: Learned to deploy and manage containerized applications using Amazon Elastic Container Service.
- ECR: Managed Docker images using Amazon Elastic Container Registry.
- EC2: Utilized Amazon EC2 instances for running Jenkins and load testing tools.
- CloudWatch: Monitored application performance and set up alerts using Amazon CloudWatch.
- IAM: Configured and managed IAM roles and policies for secure access and permissions.
- SNS: Implemented Amazon SNS for email notifications on deployment status and performance alerts.
CI/CD Pipeline Implementation:
- Jenkins: Set up a Jenkins server on AWS, installed necessary plugins, and created a Jenkins pipeline for automated deployments.
- Docker: Built, managed, and deployed Docker containers, ensuring consistent application environments.
- Automated Builds and Deployments: Configured Jenkins to automate the build and deployment process, reducing manual intervention and ensuring faster delivery.
Containerization Best Practices:
- Dockerfile Creation: Created optimized Dockerfiles for the web application.
- Docker Compose: Used Docker Compose for local development and testing of multi-container applications.
Monitoring and Load Testing:
- Apache Benchmark (ab) and wrk: Conducted load tests to measure the performance and resilience of the application under high traffic.
- CloudWatch Metrics and Alarms: Set up CloudWatch metrics and alarms to monitor CPU utilization, memory usage, and application logs.
Networking and Security:
- VPC Configuration: Created and managed VPCs, subnets, and security groups to ensure secure and isolated network environments.
- Security Groups and IAM Roles: Configured security groups to allow necessary traffic and set up IAM roles with least privilege access.
Handling Real-World Scenarios:
- Troubleshooting Deployment Issues: Resolved common deployment errors such as network configuration issues and IAM role permissions.
- Scaling and Performance Optimization: Learned to scale ECS services and optimize performance based on load testing results.
- Automated Notifications: Implemented SNS for real-time notifications on deployment status and performance issues.

Scenarios and Experiences

Scenario: Automated CI/CD Pipeline with Jenkins:
- Experience: Set up a Jenkins server on an EC2 instance, configured it with necessary plugins, and created a pipeline that automated the build, test, and deployment of a Dockerized web application to ECS. This demonstrated an understanding of continuous integration and continuous deployment best practices.
Scenario: Containerization and Deployment on ECS:
- Experience: Containerized a simple HTML web application using Docker, pushed the image to Amazon ECR, and deployed it on ECS. This involved creating and configuring task definitions, clusters, and services in ECS, showcasing knowledge of container orchestration.
Scenario: Load Testing and Monitoring:
- Experience: Conducted load testing using Apache Benchmark (ab) and wrk to simulate high traffic on the web application. Monitored application performance with CloudWatch and set up alarms to trigger notifications via SNS in case of performance degradation. This highlighted skills in performance testing and monitoring.
Scenario: Handling Deployment Errors:
- Experience: Faced and resolved issues related to ECS service creation, IAM role permissions, and network configurations. This involved troubleshooting errors such as "ResourceInitializationError" and ensuring proper network settings for ECS tasks to access ECR. This scenario demonstrated problem-solving skills and the ability to debug deployment issues.
Scenario: Secure and Scalable Network Setup:
- Experience: Configured VPCs, subnets, and security groups to create a secure and scalable network environment for the ECS cluster. This included setting up proper routing for ENIs and ensuring security group rules allowed necessary traffic. This showcased an understanding of AWS networking and security best practices.

Interview Discussion Points

Discuss the end-to-end deployment process using Jenkins and ECS, highlighting the automation of build and deployment stages.
Explain the benefits of containerization and how Docker was used to ensure consistent application environments.
Describe the monitoring setup with CloudWatch and how it helped in maintaining application performance and uptime.
Share insights on load testing results and how they influenced performance optimization and scaling decisions.
Highlight the troubleshooting steps taken to resolve deployment issues and the importance of proper IAM and network configurations.
Emphasize the importance of security in cloud deployments, mentioning the setup of VPCs, subnets, and security groups.

Command Palette

Comments

AWS Blogs

Multi-Region Deployment of Java Web Application in Azure DevOps

More from this blog

Overview

Project Description: Deploying a Simple HTML Web Application on AWS ECS Using CI/CD and Load Testing

Introduction

Objectives

Key Steps

Outcomes

Conclusion

Prerequisites

Prerequisites

Tools and AWS Services Used for the Project

Tools

AWS Services

Additional Features

Implementation Steps for SNS and Email Notifications

Deployment Steps:

Implementation in Action

Migrate the Docker application to AWS ECS Container platform

Set Up Jenkins CI/CD Pipeline

Step 1: Launch an EC2 Instance for Jenkins

Step 1: Set Up AWS Credentials in Jenkins

Step 2: Create Jenkins Pipeline

Monitoring CloudWatch

Step 1: Set Up Amazon Linux 2 Instance

Step 2: Install Load Testing Tools on Amazon Linux 2

Apache Benchmark (ab)

wrk

Step 3: Perform Load Testing

Using Apache Benchmark (ab)

Using wrk

Step 4: Monitor CPU Utilization on CloudWatch

Enable CloudWatch Monitoring for ECS Cluster

Create CloudWatch Alarms

View Metrics

Step 5: Verify and Analyze

Example CloudWatch Alarm JSON (Optional)

Project Conclusion

Overview

Key Steps and Achievements

Project Outcomes

Lessons Learned

Future Work

Conclusion

Key Takeaways from the Project

Scenarios and Experiences

Interview Discussion Points