AWS ECS Project 🌩️
Simple Web App Deployment with Jenkins CI/CD and CloudWatch Monitoring
Overview
This document provides a comprehensive guide for deploying a Docker application to AWS ECS (Elastic Container Service) and ECR (Elastic Container Registry) using Jenkins. The deployment process involves building a Docker image, pushing it to ECR, updating the ECS task definition, and deploying the updated task definition to an ECS service.
Project Description: Deploying a Simple HTML Web Application on AWS ECS Using CI/CD and Load Testing
Introduction
In this project, we explore the end-to-end process of deploying a simple HTML web application using Amazon Web Services (AWS) Elastic Container Service (ECS). The deployment is facilitated by a Continuous Integration and Continuous Deployment (CI/CD) pipeline, and the application’s performance is validated through load testing. This detailed guide covers everything from setting up the necessary infrastructure to monitoring application performance, offering insights into best practices for modern cloud-native application deployment and management.
Objectives
Set Up Jenkins and CI/CD Pipeline:
Configure Jenkins on AWS EC2.
Set up Jenkins plugins for Docker, Amazon ECR, and pipeline management.
Automate the build, test, and deployment process using Jenkins.
Dockerize the Application:
Create a Dockerfile for the HTML web application.
Build and push Docker images to Amazon ECR.
Deploy on AWS ECS:
Configure the necessary AWS infrastructure (VPC, subnets, security groups).
Deploy the Dockerized application to ECS with appropriate task definitions and service configurations.
Load Testing and Monitoring:
Perform load testing using tools like Apache Benchmark (ab) and wrk.
Monitor application performance using Amazon CloudWatch.
Set up CloudWatch alarms to alert on performance issues.
Key Steps
Jenkins Setup and CI/CD Pipeline:
Installation and configuration of Jenkins.
Setting up Jenkins plugins for a seamless CI/CD process.
Writing and configuring pipeline scripts for automated deployment.
AWS Infrastructure Configuration:
Setting up VPC, subnets, and security groups to ensure proper network configuration for ECS tasks.
Creating and attaching IAM roles with necessary permissions for ECS services.
Dockerization and Deployment:
Writing a Dockerfile for the HTML web application.
Building Docker images and pushing them to Amazon ECR.
Configuring and deploying ECS tasks and services.
Performance Testing and Monitoring:
Conducting load testing to evaluate application performance under stress.
Using Amazon CloudWatch for real-time monitoring and setting up alerts for proactive issue resolution.
Outcomes
Automated Deployment:
Successful setup of a CI/CD pipeline that automates the build and deployment process.
Reduced manual intervention and errors, speeding up the deployment cycle.
Scalable and Monitored Application:
The HTML web application is containerized and deployed on a scalable ECS cluster.
Continuous performance monitoring ensures the application remains reliable and performant under load.
Insights from Load Testing:
Valuable data on how the application handles increased traffic.
Identified and resolved potential performance bottlenecks.
Conclusion
This project provides a comprehensive guide for deploying a simple HTML web application on AWS ECS using a CI/CD pipeline. From initial setup to performance testing and monitoring, each step is detailed to ensure a thorough understanding of the deployment process. By following this guide, you will gain hands-on experience in cloud-native application deployment, continuous integration, continuous deployment, and performance monitoring, equipping you with the skills necessary for managing modern cloud-based applications effectively.
GitHub Project Repo: https://github.com/prafulpatel16/ecs-demo.git
Prerequisites
AWS Account: Ensure you have an active AWS account. Docker: Install Docker on your local machine. AWS CLI: Install and configure the AWS CLI. AWS IAM Role: Create an IAM role with permissions for ECS, ECR, and other related services. Jenkins: Install Jenkins on your local machine or use a Jenkins server. Git Repository: Set up a Git repository (e.g., GitHub, GitLab).
Create AWS IAM Role
Create ECS Task Role Name: ecsTaskExecutionRole
Create JSON Policy: EC2ContainerRegistryReadOnly
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:BatchGetImage",
"ecr:DescribeRepositories",
"ecr:GetDownloadUrlForLayer",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:GetRepositoryPolicy",
"ecr:DescribeImageScanFindings",
"ecr:ListTagsForResource",
"ecr:DescribeRegistry",
"ecr:GetAuthorizationToken"
],
"Resource": "*"
}
]
}
Prerequisites
AWS Account: Ensure you have an active AWS account.
Docker: Install Docker on your local machine.
AWS CLI: Install and configure the AWS CLI.
Code Repository: Set up a Git repository (e.g., GitHub, GitLab).
CI/CD Tool: Use a CI/CD tool like GitHub Actions, GitLab CI, or AWS CodePipeline.
Tools and AWS Services Used for the Project
Tools
Jenkins
Description: An open-source automation server used to automate the building, testing, and deployment of applications.
Usage: Set up CI/CD pipelines to automate the build and deployment process.
Docker
Description: A platform for developing, shipping, and running applications in containers.
Usage: Containerize the HTML web application.
Apache Benchmark (ab)
Description: A tool for benchmarking the performance of HTTP web servers.
Usage: Perform load testing on the deployed application.
AWS Services
Amazon Elastic Container Service (ECS)
Description: A fully managed container orchestration service.
Usage: Deploy and manage the containerized HTML web application.
Amazon Elastic Container Registry (ECR)
Description: A fully managed Docker container registry that makes it easy to store, manage, and deploy Docker container images.
Usage: Store and manage Docker images for deployment on ECS.
Amazon Elastic Compute Cloud (EC2)
Description: A web service that provides resizable compute capacity in the cloud.
Usage: Host Jenkins for setting up the CI/CD pipeline.
Amazon Virtual Private Cloud (VPC)
Description: A service that lets you launch AWS resources in a logically isolated virtual network.
Usage: Create a secure and isolated network environment for ECS tasks and services.
Amazon CloudWatch
Description: A monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers.
Usage: Monitor application performance, set up alarms, and gain insights into resource utilization.
AWS Identity and Access Management (IAM)
Description: A web service that helps you securely control access to AWS services and resources.
Usage: Manage permissions and roles for ECS tasks and services, ensuring secure access control.
AWS Secrets Manager
Description: A service to help you protect access to your applications, services, and IT resources without the upfront cost and maintenance of hardware security modules (HSMs).
Usage: Manage and retrieve secrets such as database credentials securely.
Amazon Route 53
Description: A scalable and highly available Domain Name System (DNS) web service.
Usage: Route traffic to the application deployed on ECS.
Amazon Simple Notification Service (SNS)
Description: A fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
Usage: Send email notifications about the status of the ECS tasks and alarms set in CloudWatch.
Additional Features
Email Notifications Using SNS
Description: Set up Amazon SNS to send email notifications for important events or alarms related to the ECS deployment.
Usage: Create an SNS topic, subscribe an email endpoint to the topic, and configure CloudWatch to send notifications to the SNS topic.
Implementation Steps for SNS and Email Notifications
Set Up SNS Topic and Email Subscription
Create SNS Topic:
aws sns create-topic --name ecs-deployment-notifications
Subscribe Email to SNS Topic:
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications --protocol email --notification-endpoint your-email@example.com
Configure CloudWatch Alarms to Send Notifications
Create CloudWatch Alarm:
aws cloudwatch put-metric-alarm --alarm-name "HighCPUUtilization" --metric-name "CPUUtilization" --namespace "AWS/ECS" --statistic "Average" --period 300 --threshold 80 --comparison-operator "GreaterThanThreshold" --dimensions Name=ClusterName,Value=your-cluster-name --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications
By integrating Amazon SNS and email notifications, you ensure that you receive timely updates on the status of your ECS deployments and any critical alerts, thereby enhancing the observability and reliability of your deployment process.
Deployment Steps:
1. Prepare the HTML Web App
2. Dockerize the Application
3. Create a Docker Repository on AWS ECR
4. Push the Docker Image to ECR
5. Create ECS Cluster and Task Definition
6. Create ECS Service
7. Set Up CI/CD Pipeline
8. Monitor with CloudWatch
9. Test and Validate
Implementation in Action
1. Prepare the HTML Web App
2. Dockerize the Application
# Use an official nginx image as the base image
FROM nginx:alpine
# Copy the HTML file to the nginx directory
COPY index.html /usr/share/nginx/html
# Expose port 80
EXPOSE 80
# Start nginx when the container launches
CMD ["nginx", "-g", "daemon off;"]
Build Docker Image: Build docker image in local
docker build -t simple-html-web-app .
docker image build successful
docker images
Let's the docker image locally
Start the docker image 'simple-html-web-app'
docker run -d -p 8080:80 simple-html-web-app:latest
Verify the docker image web app into Browser
Migrate the Docker application to AWS ECS Container platform
3. Create a Docker Repository on AWS ECR
Create ECR Repository:
aws ecr create-repository --repository-name simple-html-web-app
Login to ECR:
aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com
4. Push the Docker Image to ECR
docker tag simple-html-web-app:latest <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest
docker push <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest
5. Create ECS Cluster and Task Definition
Create ECS Cluster:
aws ecs create-cluster --cluster-name simple-html-web-app-cluster
Create Task Definition:
taskdef.json
{
"family": "simple-html-web-app-task",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "simple-html-web-app-container",
"image": "<your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest",
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"memory": 512,
"cpu": 256
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::<your-aws-account-id>:role/ecsTaskExecutionRole"
}
Register Task Definition:
aws ecs register-task-definition --cli-input-json file://task-definition.json
6. Create ECS Service
Create Service:
from terminal using aws cli
aws ecs create-service \
--cluster simple-html-web-app-cluster \
--service-name simple-html-web-app-service \
--task-definition simple-html-web-app-task \
--desired-count 1 \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[<your-subnet-id>],securityGroups=[<your-security-group-id>],assignPublicIp=ENABLED}"
from AWS Console
Networking: It is going to leverage default VPC and it's default security group at the moment
Create new Application Load Balancer
New ECS Service created
Set Up Jenkins CI/CD Pipeline
Jenkins installation
Create EC2 machine and Jenkins Server and install Jenkins
https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/
Part 1: Set Up Jenkins on AWS
Step 1: Launch an EC2 Instance for Jenkins
Sign in to AWS Management Console.
Launch EC2 Instance:
Go to Services and select EC2.
Click on Launch Instance.
Choose AMI:
- Select an Amazon Linux 2 AMI (or Ubuntu, if preferred).
Choose Instance Type:
- Select
t2.medium
(or another type based on your needs).
- Select
Configure Instance:
Configure VPC and Subnet.
Enable Auto-assign Public IP.
Add Storage:
- Configure storage as needed (8GB or more).
Add Tags:
- Add tags for easy identification (e.g., Key:
Name
, Value:Jenkins
).
- Add tags for easy identification (e.g., Key:
Configure Security Group:
Create a new security group with the following inbound rules:
HTTP: Port 80, Source: 0.0.0.0/0
Custom TCP Rule: Port 8080, Source: 0.0.0.0/0
SSH: Port 22, Source: 0.0.0.0/0
Review and Launch:
Review your settings and launch the instance.
Select or create a key pair for SSH access.
Connect to the EC2 Instance:
- Use SSH to connect to your EC2 instance.
shCopy codessh -i "your-key-pair.pem" ec2-user@your-ec2-public-dns
Install Java:
shCopy codesudo yum update -y sudo amazon-linux-extras install java-openjdk11 -y
Add Jenkins Repository and Install Jenkins:
shCopy codesudo wget -O /etc/yum.repos.d/jenkins.repo \ https://pkg.jenkins.io/redhat-stable/jenkins.repo sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key sudo yum install jenkins -y
Start Jenkins:
shCopy codesudo systemctl start jenkins sudo systemctl enable jenkins
Open Jenkins in Browser:
Navigate to
http://your-ec2-public-dns:8080
in your browser.Retrieve the initial admin password:
shCopy codesudo cat /var/lib/jenkins/secrets/initialAdminPassword
Unlock Jenkins:
Paste the retrieved password to unlock Jenkins.
Install suggested plugins during setup.
Configure Jenkins Credentials for AWS integration
Global
Add Credentials
Create a jenkins user in AWS IAM role with access key and id and provide those credentials in jenkins
Part 2: Install Required Jenkins Plugins
Install Plugins:
Go to
Manage Jenkins
>Manage Plugins
>Available
.Search and install the following plugins:
Docker Pipeline
Amazon ECR
Pipeline
Git
Part 3: Configure Jenkins Pipeline for ECS
Step 1: Set Up AWS Credentials in Jenkins
Create AWS IAM User for Jenkins:
Go to IAM > Users > Add user.
Username:
jenkins-user
, Access type:Programmatic access
.Attach existing policies directly:
AmazonEC2ContainerRegistryFullAccess
,AmazonECS_FullAccess
,AmazonS3FullAccess
.Download the Access key ID and Secret access key.
Configure Credentials in Jenkins:
Go to
Manage Jenkins
>Manage Credentials
.Under the appropriate domain (e.g.,
(global)
), add a new AWS Credentials:Kind: AWS Credentials
Access Key ID and Secret Access Key from the IAM user.
Step 2: Create Jenkins Pipeline
Create a New Pipeline Job:
Go to
Jenkins
>New Item
.Enter a name (e.g.,
ECS-Deploy-Pipeline
), select Pipeline, and click OK.
Configure Pipeline Script:
In the Pipeline section, choose Pipeline script.
Enter the following example script:
Create a Pipline CICD
pipeline {
agent any
parameters {
string(name: 'IMAGE_TAG', defaultValue: 'latest', description: 'Docker image tag')
}
environment {
AWS_ACCOUNT_ID = '913151559'
AWS_REGION = 'us-east-1'
ECR_REPO = 'simple-html-web-app'
ECR_REPO_URI = '9313151559.dkr.ecr.us-east-1.amazonaws.com/simple-html-web-app'
REGISTRY_CREDENTIAL = 'aws-credentials-id' // Ensure this matches the credentials ID in Jenkins
IMAGE_TAG = "latest-${env.BUILD_NUMBER}" // Unique tag for each build
ECS_CLUSTER = 'simple-html-web-app-cluster'
ECS_SERVICE = 'ecs-demo-srv'
TASK_DEFINITION_FAMILY = 'ecs-task-def'
}
stages {
stage('Test Credentials') {
steps {
script {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
sh 'aws sts get-caller-identity'
}
}
}
}
stage('Clone Git Repository') {
steps {
checkout([$class: 'GitSCM',
branches: [[name: '*/master']],
doGenerateSubmoduleConfigurations: false,
extensions: [],
submoduleCfg: [],
userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/prafulpatel16/ecs-demo.git']]
])
}
}
stage('Build and Push Docker Image') {
steps {
script {
// Build and tag Docker image with specified tag
sh "docker build -t ${ECR_REPO_URI}:${params.IMAGE_TAG} ."
// Login to AWS ECR
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
sh """
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REPO_URI}
"""
}
// Push Docker image to ECR
sh "docker push ${ECR_REPO_URI}:${params.IMAGE_TAG}"
}
}
}
stage('Update ECS Task Definition') {
steps {
script {
// Load and update task definition JSON
def taskDefinition = readFile 'taskdef.json'
taskDefinition = taskDefinition.replace("REPLACE_WITH_IMAGE_TAG", "${ECR_REPO_URI}:${params.IMAGE_TAG}")
// Write updated task definition to file
writeFile file: 'taskdef.json', text: taskDefinition
// Register updated task definition and capture the revision number
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
def registerOutput = sh(script: "aws ecs register-task-definition --cli-input-json file://taskdef.json", returnStdout: true).trim()
echo "Register Output: ${registerOutput}"
def json = readJSON text: registerOutput
def taskDefinitionArn = json.taskDefinition.taskDefinitionArn
echo "Task Definition ARN: ${taskDefinitionArn}"
def taskDefinitionRevision = taskDefinitionArn.tokenize(':').last() // Extract revision number
echo "Task Definition Revision: ${taskDefinitionRevision}"
// Save the new task definition revision for later use
env.TASK_DEFINITION_REVISION = taskDefinitionRevision
}
}
}
}
stage('Deploy ECS Service') {
steps {
script {
// Update ECS service with the new task definition revision
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
sh """
aws ecs update-service --cluster ${ECS_CLUSTER} --service ${ECS_SERVICE} --task-definition ${TASK_DEFINITION_FAMILY}:${env.TASK_DEFINITION_REVISION} --force-new-deployment
"""
}
}
}
}
}
}
CICD Pipeline is ready to build and deploy
Before Deployment verify the ECR repo and ECS task definition
ECR
ECS Task Definition
Let's build the code and deploy
Build Manual
Build and deploy successful
Access Web application
Hit the ALB url: http://ecs-alb-611614342.us-east-1.elb.amazonaws.com/
Verify post deployment that the new task definition has been created
Let's Change and update the code to verify that the new changes successfully apply within the CICD pipeline
Observe that on the web application there is text "Demo ECS" needs to be removed from the code and new code needs to be deployed through CICD
Go to Local VS code and remove the text
After removal the text
Commit the changes and push to the GitHub repo
git push
Expectation: New changes should be updated successfully on the web application
So if we look at the ECS Cluster definition there is 'ecs-task-def:28' which is curently running with the old web application and once the new changes is deployed it should be new 'ecs-task-def:29' should be created with the new changes.
Now let's build the code and push the changes through Jenkins CICD
Build#18
Build Successful
new 'ecs-task-def:29' deploying
deployed
Let's verify the web application has updated the changes successfully
removed text "demo-ecs"
Monitoring CloudWatch
Certainly! Here’s a comprehensive guide to performing a CPU load test on an ECS cluster web app and monitoring CPU utilization on CloudWatch using Apache Benchmark (ab) and wrk from an Amazon Linux 2 AMI:
Step 1: Set Up Amazon Linux 2 Instance
Launch an Amazon Linux 2 Instance:
Open the EC2 Dashboard in the AWS Management Console.
Launch a new instance and select Amazon Linux 2 AMI.
Choose an instance type (e.g., t2.micro).
Configure instance details, add storage, and configure security groups to allow SSH access.
Launch the instance and connect to it via SSH.
Step 2: Install Load Testing Tools on Amazon Linux 2
Apache Benchmark (ab)
Install Apache Benchmark:
sudo yum update -y sudo yum install httpd-tools -y
wrk
Install wrk:
sudo yum install -y git gcc git clone https://github.com/wg/wrk.git cd wrk make sudo cp wrk /usr/local/bin
Step 3: Perform Load Testing
Using Apache Benchmark (ab)
Run a Basic Load Test:
ab -n 1000 -c 50 http://your-alb-url/
-n 1000
: Number of requests to perform.-c 50
: Number of multiple requests to perform at a time.Replace
http://your-alb-url/
with your actual ALB URL.
Increase the Load:
ab -n 10000 -c 200 http://your-alb-url/
Using wrk
Run a Basic Load Test:
wrk -t12 -c400 -d30s http://your-alb-url/
-t12
: Number of threads to use.-c400
: Number of connections to open.-d30s
: Duration of the test (30 seconds).Replace
http://your-alb-url/
with your actual ALB URL.
Step 4: Monitor CPU Utilization on CloudWatch
Enable CloudWatch Monitoring for ECS Cluster
Ensure CloudWatch Monitoring is Enabled:
Navigate to the ECS Cluster in the AWS Management Console.
Go to the "Monitoring" tab and ensure CloudWatch metrics are enabled.
Create CloudWatch Alarms
Create an Alarm for CPU Utilization:
Open the CloudWatch Dashboard in the AWS Management Console.
Click on "Alarms" > "Create Alarm".
Select the ECS cluster's CPUUtilization metric.
Set the threshold (e.g., CPU utilization > 80% for 5 minutes).
Configure actions (e.g., send an SNS notification).
Review and create the alarm.
View Metrics
View CPU Utilization Metrics:
In the CloudWatch Dashboard, navigate to "Metrics".
Select "ECS" and find the CPU utilization metrics for your cluster and services.
Step 5: Verify and Analyze
Verify Load Test Results:
Check the output of
ab
orwrk
for request statistics, including requests per second, mean response time, and more.Ensure your web application is handling the load as expected.
Analyze CloudWatch Metrics:
Go to the CloudWatch Dashboard and check the CPU utilization metrics.
Ensure that the ECS tasks are scaling properly based on the load and the alarms are triggering as expected.
Example CloudWatch Alarm JSON (Optional)
If you prefer to create the CloudWatch alarm using AWS CLI, here is an example JSON configuration:
{
"AlarmName": "ECS-CPU-Utilization-High",
"AlarmDescription": "Alarm when ECS CPU utilization exceeds 80%",
"ActionsEnabled": true,
"OKActions": [],
"AlarmActions": [
"arn:aws:sns:us-east-1:123456789012:MySNSTopic"
],
"MetricName": "CPUUtilization",
"Namespace": "AWS/ECS",
"Statistic": "Average",
"Dimensions": [
{
"Name": "ClusterName",
"Value": "your-ecs-cluster-name"
},
{
"Name": "ServiceName",
"Value": "your-ecs-service-name"
}
],
"Period": 300,
"EvaluationPeriods": 1,
"Threshold": 80.0,
"ComparisonOperator": "GreaterThanThreshold"
}
You can use the AWS CLI to create the alarm with the above configuration:
aws cloudwatch put-metric-alarm --cli-input-json file://alarm.json
Replace [file://alarm.json
](file://alarm.json) with the path to your JSON file.
By following these steps, you can perform load testing on your ECS cluster web application and monitor its CPU utilization using CloudWatch.
Create Alarm
Select Metric
Increase the Load:
SSH to the Load test machine
ab -n 10000 -c 200 http://your-alb-url/
Hit the ALB URL 10 to 20 times and observe the load
Observer the CPU Load
ALarm Triggered
Email Notification triggered
High CPU Utlilization
Project Conclusion
Overview
The objective of this project was to deploy a simple HTML web application on AWS ECS using a CI/CD pipeline and to ensure the application’s performance under load through load testing and monitoring. This comprehensive exercise demonstrated the full lifecycle of a cloud-native application, from development and deployment to performance monitoring and scaling.
Key Steps and Achievements
Setting Up Jenkins and CI/CD Pipeline:
Installed and configured Jenkins on an AWS EC2 instance.
Set up necessary Jenkins plugins for Docker, Amazon ECR, and pipeline management.
Created a Jenkins pipeline script to build, push Docker images to ECR, and deploy to ECS.
Dockerization and Deployment to AWS ECS:
Created a Dockerfile for the simple HTML web application.
Built Docker images and pushed them to Amazon ECR.
Deployed the application to AWS ECS with appropriate task definitions and service configurations.
Infrastructure Setup on AWS:
Configured necessary VPC, subnets, and security groups to ensure the ECS tasks had proper networking configurations.
Created IAM roles and policies to grant necessary permissions for ECS tasks and services.
Load Testing and Monitoring:
Performed load testing using Apache Benchmark (ab) and wrk to simulate user traffic and evaluate the application's performance.
Monitored ECS cluster and service performance using Amazon CloudWatch.
Configured CloudWatch alarms to alert when CPU utilization thresholds were exceeded.
Troubleshooting and Optimization:
Resolved common issues such as task network configuration errors and IAM role permissions.
Tuned ECS service and task definitions for optimal performance under load.
Project Outcomes
Successful Deployment:
The HTML web application was successfully containerized and deployed on AWS ECS.
The CI/CD pipeline automated the process of building, testing, and deploying the application, ensuring efficient and reliable deployments.
Effective Load Testing:
Load tests provided valuable insights into the application's performance and scalability.
Identified and mitigated potential bottlenecks, ensuring the application could handle increased traffic.
Robust Monitoring and Alerts:
CloudWatch metrics and alarms enabled proactive monitoring of the application's performance.
Ensured timely alerts and response to any performance degradation or failures.
Lessons Learned
Automation is Key:
- Automating the deployment process using Jenkins and ECS significantly reduces manual errors and speeds up the deployment process.
Importance of Monitoring:
- Continuous monitoring and proactive alerting are crucial for maintaining application performance and availability.
Thorough Testing:
- Load testing is essential to understand how the application performs under various traffic conditions and to ensure it can scale appropriately.
Future Work
Advanced CI/CD Features:
- Implementing advanced CI/CD features such as blue-green deployments or canary releases to minimize downtime and reduce risk during updates.
Enhanced Monitoring:
- Integrating more sophisticated monitoring and logging tools such as AWS X-Ray for distributed tracing and deeper insights into application performance.
Security Enhancements:
- Implementing more robust security measures, including using AWS Secrets Manager for managing sensitive information and enhancing IAM policies.
Conclusion
This project demonstrated the end-to-end process of deploying a web application on AWS ECS, from setting up the CI/CD pipeline to ensuring the application’s performance under load. Through this exercise, key skills in cloud infrastructure management, continuous integration, continuous deployment, and performance monitoring were reinforced, providing a solid foundation for managing and scaling cloud-native applications effectively.
Key Takeaways from the Project
Hands-On Experience with AWS Services:
ECS: Learned to deploy and manage containerized applications using Amazon Elastic Container Service.
ECR: Managed Docker images using Amazon Elastic Container Registry.
EC2: Utilized Amazon EC2 instances for running Jenkins and load testing tools.
CloudWatch: Monitored application performance and set up alerts using Amazon CloudWatch.
IAM: Configured and managed IAM roles and policies for secure access and permissions.
SNS: Implemented Amazon SNS for email notifications on deployment status and performance alerts.
CI/CD Pipeline Implementation:
Jenkins: Set up a Jenkins server on AWS, installed necessary plugins, and created a Jenkins pipeline for automated deployments.
Docker: Built, managed, and deployed Docker containers, ensuring consistent application environments.
Automated Builds and Deployments: Configured Jenkins to automate the build and deployment process, reducing manual intervention and ensuring faster delivery.
Containerization Best Practices:
Dockerfile Creation: Created optimized Dockerfiles for the web application.
Docker Compose: Used Docker Compose for local development and testing of multi-container applications.
Monitoring and Load Testing:
Apache Benchmark (ab) and wrk: Conducted load tests to measure the performance and resilience of the application under high traffic.
CloudWatch Metrics and Alarms: Set up CloudWatch metrics and alarms to monitor CPU utilization, memory usage, and application logs.
Networking and Security:
VPC Configuration: Created and managed VPCs, subnets, and security groups to ensure secure and isolated network environments.
Security Groups and IAM Roles: Configured security groups to allow necessary traffic and set up IAM roles with least privilege access.
Handling Real-World Scenarios:
Troubleshooting Deployment Issues: Resolved common deployment errors such as network configuration issues and IAM role permissions.
Scaling and Performance Optimization: Learned to scale ECS services and optimize performance based on load testing results.
Automated Notifications: Implemented SNS for real-time notifications on deployment status and performance issues.
Scenarios and Experiences
Scenario: Automated CI/CD Pipeline with Jenkins:
- Experience: Set up a Jenkins server on an EC2 instance, configured it with necessary plugins, and created a pipeline that automated the build, test, and deployment of a Dockerized web application to ECS. This demonstrated an understanding of continuous integration and continuous deployment best practices.
Scenario: Containerization and Deployment on ECS:
- Experience: Containerized a simple HTML web application using Docker, pushed the image to Amazon ECR, and deployed it on ECS. This involved creating and configuring task definitions, clusters, and services in ECS, showcasing knowledge of container orchestration.
Scenario: Load Testing and Monitoring:
- Experience: Conducted load testing using Apache Benchmark (ab) and wrk to simulate high traffic on the web application. Monitored application performance with CloudWatch and set up alarms to trigger notifications via SNS in case of performance degradation. This highlighted skills in performance testing and monitoring.
Scenario: Handling Deployment Errors:
- Experience: Faced and resolved issues related to ECS service creation, IAM role permissions, and network configurations. This involved troubleshooting errors such as "ResourceInitializationError" and ensuring proper network settings for ECS tasks to access ECR. This scenario demonstrated problem-solving skills and the ability to debug deployment issues.
Scenario: Secure and Scalable Network Setup:
- Experience: Configured VPCs, subnets, and security groups to create a secure and scalable network environment for the ECS cluster. This included setting up proper routing for ENIs and ensuring security group rules allowed necessary traffic. This showcased an understanding of AWS networking and security best practices.
Interview Discussion Points
Discuss the end-to-end deployment process using Jenkins and ECS, highlighting the automation of build and deployment stages.
Explain the benefits of containerization and how Docker was used to ensure consistent application environments.
Describe the monitoring setup with CloudWatch and how it helped in maintaining application performance and uptime.
Share insights on load testing results and how they influenced performance optimization and scaling decisions.
Highlight the troubleshooting steps taken to resolve deployment issues and the importance of proper IAM and network configurations.
Emphasize the importance of security in cloud deployments, mentioning the setup of VPCs, subnets, and security groups.