Welcome to my comprehensive guide on building a Serverless CRUD Application using AWS! In this post, we'll dive deep into how to create and deploy a fully serverless, scalable, and cost-effective CRUD system using a variety of AWS services. Whether you're a developer eager to explore serverless architectures or looking for a rapid prototyping solution, this guide is for you! 😊

Follow the GitHub Documentation and code for full implementation details:
🔹 GitHub Repo: serverless-crud-app

Table of Contents

1. About the Project

2. Architecture

3. Project Structure

4. Prerequisites & Setup

5. Documentations

6. Detailed Workflow

7. CRUD Operations

8. Deployment & Configuration

9. Local Testing

10. AWS Services Used

11. Contributing

12. License

13. Author

14. Additional Resources

About the Project 📝

The Serverless CRUD App is a complete backend and frontend solution that leverages a fully serverless architecture to manage Create, Read, Update, and Delete operations. Built with AWS SAM, CloudFormation, and a structured backend API, this project seamlessly integrates with a simple HTML/JavaScript frontend hosted on Amazon S3.

Why build a Serverless CRUD App?

• Scalability: Automatically scales based on demand.

• Cost-Effective: Pay only for what you use.

• Simplicity: Focus on business logic instead of server maintenance.

• High Availability: AWS services ensure robust, fault-tolerant operations.

When is this useful?

• Rapid prototyping of applications.

• Developing low to medium traffic apps.

• Optimizing costs for intermittent workloads.

Architectural Flow🏗️

The app is built on a modern AWS serverless stack. Here's an architectural overview:

Key Components:

• Frontend (S3): Hosts the static HTML, CSS, and JS.

• API Gateway: Exposes RESTful endpoints for CRUD operations.

• Lambda Functions: Execute CRUD logic and handle API requests.

• DynamoDB: NoSQL database storing items (e.g., id, name, age).

Frontend UI

Project Structure 📁

serverless-crud-app/
├── backend/
│   ├── create/         # Lambda function for creating resources
│   ├── read/           # Lambda function for reading a single resource
│   ├── update/         # Lambda function for updating resources
│   ├── delete/         # Lambda function for deleting resources
│   ├── list/           # Lambda function for listing resources
│   └── documentations/ # Detailed API and infrastructure docs
├── frontend/
│   └── index.html      # Static frontend UI
├── pipelines/          # CI/CD pipeline configurations
├── cloudformation.yaml # CloudFormation/SAM stack configuration
├── template.yaml       # AWS SAM template definition
└── README.md           # Project documentation

• backend/: Contains all Lambda function code and documentation.

• frontend/: Hosts the static UI assets.

• pipelines/: Holds configurations for automated CI/CD deployments.

• cloudformation.yaml / template.yaml: Define the AWS infrastructure as code.

• README.md: Provides an overview and detailed documentation for the project.

Prerequisites & Setup 🛠️

Before you begin, ensure you have:

• AWS Account: Required to deploy AWS resources.

• AWS CLI: Installation Guide and configure with aws configure.

• AWS SAM CLI / CloudFormation: For packaging and deploying the serverless application.

• Node.js / Python: For frontend and backend development/testing.

Documentations 📜

Additional documentation files can be found in the documentations/ directory:

• 0.0 Pre-requisite

• 0.1 IAM Roles

• 1. API Documentation

• 2. API Testing

• 3. App Functionality

• 4. Frontend Documentation

• 5. SAM Infra Documentation

Detailed Workflow 🔄

1. User Accesses the Frontend:
   The user opens the static site hosted on Amazon S3 (optionally via CloudFront).

2. User Chooses a CRUD Operation:

   • Create: Fill out ID, Name, and Age fields.

   • Read: Enter an ID to fetch details.

   • Update: Enter an ID along with fields to update.

   • Delete: Enter an ID to remove an item.

   • List: Retrieve all items from the DynamoDB table.

3. Request Routing via API Gateway:
   The frontend sends an HTTP request to API Gateway, which routes it to the appropriate Lambda function.

4. Lambda Function Processing:
   The Lambda function performs the CRUD operation on the DynamoDB table and returns a JSON response.

5. Response Displayed on the Frontend:
   The result (success or error) is displayed to the user in the UI.

CRUD Operations ⚙️

1. Create (POST /create):

   • Frontend: Sends an array of items (e.g., [{ id, name, age }, ...]).

   • Lambda: Inserts each item into DynamoDB using put_item.

2. Read (GET /read?id=123):

   • Frontend: Appends ?id=123 to the URL.

   • Lambda: Retrieves an item using get_item(Key={"id": ...}).

3. Update (PUT /update):

   • Frontend: Sends a JSON payload with id and fields to update.

   • Lambda: Uses update_item with dynamic update expressions.

4. Delete (DELETE /delete):

   • Frontend: Sends a JSON payload with "Key": { "id": "..." }.

   • Lambda: Calls delete_item(Key={"id": ...}).

5. List (GET /list):

   • Frontend: Issues a GET request to retrieve all items.

   • Lambda: Scans or queries the DynamoDB table to list items.

Deployment & Configuration 🚀

1. CloudFormation / AWS SAM:
   Run:

    sam build && sam deploy --guided
   

   This command deploys your backend resources, including Lambda functions, DynamoDB, and API Gateway.

2. S3 Frontend Hosting:

   • Upload your frontend/index.html (and assets) to an S3 bucket.

   • Enable Static Website Hosting on the bucket.

   • Optionally, configure CloudFront for better global distribution.

3. Environment Variables:
   Set TABLE_NAME in your Lambda configuration to the name of your DynamoDB table.

4. CORS Configuration:
   Ensure each Lambda response includes Access-Control-Allow-Origin: *. API Gateway should be configured with Lambda Proxy Integration to pass these headers through.

AWS Services Used 🌐

This project leverages several key AWS services:

• AWS S3: For hosting the static frontend.

• Amazon API Gateway: To expose RESTful endpoints.

• AWS Lambda: For executing backend CRUD logic.

• AWS CloudFormation / SAM: For defining and deploying infrastructure as code.

• Amazon DynamoDB: As a scalable NoSQL database.

• AWS CloudWatch: For logging and monitoring.

• AWS X-Ray: For tracing and debugging Lambda invocations.

• AWS IAM: To securely manage permissions and roles.

• AWS CloudFront: For global content delivery (optional).

• AWS Cognito: (Optional) For user authentication and authorization if needed.
  POSTMAN

Features ✨

• 🛠️ Serverless Backend: Leverages AWS Lambda to handle CRUD operations.

• 📜 Infrastructure as Code: Built with AWS SAM and CloudFormation templates.

• 🎨 Modern Frontend UI: A simple HTML/JavaScript interface for interacting with the backend.

• 🚀 CI/CD Pipelines: Automated deployments via AWS CodePipeline or GitHub Actions.

• 📖 Comprehensive Documentation: Detailed API docs, testing guides, and architecture overviews.

Local Testing 🧪

1. Using Postman:

   • Create: POST /create with JSON body: {"items": [{"id":"123","name":"Test","age":25}]}

   • Read: GET /read?id=123

   • Update: PUT /update with JSON body: {"id":"123","name":"NewName","age":30}

   • Delete: DELETE /delete with JSON body: {"Key":{"id":"123"}}

   • List: GET /list

2. CloudWatch Logs:
   Use CloudWatch to monitor Lambda logs and debug issues.

Contributing 🤝

Contributions are welcome! Here’s how you can contribute:

1. Fork & Clone: Fork the repository on GitHub and clone it locally.

2. Create a Feature Branch:

    git checkout -b feature/my-awesome-feature
   

3. Commit & Push: Make your changes, commit, and push them to your fork.

4. Open a Pull Request: Provide detailed descriptions of your changes and open a PR against the main branch.

For additional guidelines, please refer to the CONTRIBUTING.md file.

License 📄

This project is licensed under the MIT License.

Author 👨‍💻

Created and maintained by Praful Patel.

• Praful's GitHub

• Praful's Blog

Additional Resources 📚

• AWS Lambda Documentation

• Amazon API Gateway Documentation

• Amazon DynamoDB Documentation

• AWS SAM Documentation

Happy Building!
If you have any questions or need further details, feel free to open an issue on GitHub or reach out directly.

Migrating databases from on-premises to AWS is a critical step in cloud adoption. AWS Database Migration Service (DMS) helps streamline this process by providing a secure, scalable, and automated solution for full-load and ongoing replication of databases.

Following Phase 1 - Application Discovery & TCO Analysis, we now move to Phase 2: AWS Database Migration to efficiently transition MySQL databases from an on-premises setup to Amazon RDS (Relational Database Service).

📌 Key Objectives of Phase 2

✅ Provision AWS RDS as the target database.
✅ Create a DMS Replication Instance to facilitate data migration.
✅ Configure Source & Target DMS Endpoints for secure connections.
✅ Enable Binary Logs in MySQL for ongoing replication.
✅ Execute AWS DMS Migration Task for database replication.

By the end of this phase, the on-prem MySQL database will be fully migrated to AWS RDS, ensuring data integrity, security, and scalability.

📑 Follow the GitHub Documentation for Full Details

🔹 GitHub Repo: AWS Migration Project
📌 Phase 2: AWS Database Migration
➡️ Pre-Requisite: GitHub Link
➡️ Database Migration: GitHub Link
➡️ Troubleshooting: GitHub Link

🔹 🎥 Watch the Video Tutorial on YouTube: Click Here

🌍 On-Premises Database Overview

Before migration, let’s understand the existing on-prem database setup:

📌 Database Server Details
🔹 Database: MySQL 5.7
🔹 Operating System: Ubuntu 24.04 LTS
🔹 Storage: 8 GB SSD
🔹 Tables: obbs.tbladmin, obbs.tblbooking, obbs.tblcontact, etc.
🔹 Replication Type: Full Load + Ongoing Replication (CDC)

💡 Challenges with On-Premises Databases:
❌ High Maintenance Costs – Requires manual upgrades and monitoring.
❌ Limited Scalability – Hard to scale with increasing data load.
❌ Backup & Recovery Issues – No automated snapshot capabilities.

✅ Why AWS RDS?
✔️ Fully managed database with automated backups.
✔️ High availability with Multi-AZ deployment.
✔️ Auto-scaling capabilities for dynamic workloads.

🏗 Step 1: Create Target RDS Database in AWS

📌 The first step is to provision an Amazon RDS MySQL instance as the target database.

🛠 Steps to Create RDS Instance

1️⃣ Go to AWS Console → Amazon RDS → Create Database
2️⃣ Select MySQL as Engine Type
3️⃣ Choose Multi-AZ Deployment for High Availability
4️⃣ Set Master Username & Password
5️⃣ Configure Security Groups for Database Access
6️⃣ Enable Automated Backups & Monitoring
7️⃣ Click on Create Database

📌 Once the RDS instance is running, note the endpoint.
✅ Example: database-1.cpioo8iee1me.us-west-2.rds.amazonaws.com

🚀 Step 2: Create AWS DMS Replication Instance

📌 AWS DMS Replication Instance acts as a bridge to replicate data from the source MySQL database to Amazon RDS.

🛠 Steps to Create a DMS Replication Instance

1️⃣ Go to AWS DMS Console → Replication Instances → Create Replication Instance
2️⃣ Choose Instance Type: dms.t3.medium (for moderate workloads)
3️⃣ Set Storage: 100 GB (adjust based on database size)
4️⃣ Select VPC & Security Groups: Ensure RDS and On-Prem Server are accessible
5️⃣ Click Create Replication Instance

📌 Once the replication instance is available, proceed to configuring endpoints.

🔄 Step 3: Configure Source & Target Endpoints

🔹 Source Endpoint (On-Prem MySQL)

1️⃣ Go to AWS DMS Console → Endpoints → Create Endpoint
2️⃣ Choose Source Type: MySQL
3️⃣ Enter Source Database Endpoint: <on-prem-db-IP>:3306
4️⃣ Enter Username & Password
5️⃣ Test Connection → If Successful, Save Endpoint

🔹 Target Endpoint (Amazon RDS MySQL)

1️⃣ Go to AWS DMS Console → Endpoints → Create Endpoint
2️⃣ Choose Target Type: MySQL
3️⃣ Enter Amazon RDS Endpoint: database-1.cpioo8iee1me.us-west-2.rds.amazonaws.com
4️⃣ Enter Username & Password
5️⃣ Test Connection → If Successful, Save Endpoint

✅ With both endpoints successfully configured, we move to DMS Migration Task.

🚀 Step 4: Execute DMS Migration Task

📌 Create a migration task in AWS DMS to transfer data from On-Prem MySQL to AWS RDS.

🛠 Steps to Create DMS Migration Task

1️⃣ Go to AWS DMS Console → Database Migration Tasks → Create Task
2️⃣ Select Source & Target Endpoints
3️⃣ Choose Replication Type: Full Load + Ongoing Replication (CDC)
4️⃣ Enable Logging & CloudWatch Monitoring
5️⃣ Click Start Migration Task

📌 DMS will now start migrating the data. Check CloudWatch logs for progress.

Status: Created

Status: Load Complete

✅ Step 5: Testing & Validation

📌 Once migration is complete, validate the database in AWS RDS.

🛠 Steps to Verify Data Migration

1️⃣ Connect to AWS RDS using MySQL Workbench or CLI.

mysql -h database-1.cpioo8iee1me.us-west-2.rds.amazonaws.com -u admin -p

2️⃣ Check database and tables.

SHOW DATABASES;
USE obbs;
SHOW TABLES;

3️⃣ Verify Data Consistency

SELECT COUNT(*) FROM obbs.tbladmin;

✅ Data successfully migrated from on-prem to AWS RDS! 🎉

🎯 Summary of Phase 2

✅ Provisioned AWS RDS MySQL as the Target Database.
✅ Created AWS DMS Replication Instance for Data Transfer.
✅ Configured Source & Target Endpoints for Secure Migration.
✅ Executed DMS Migration Task for Full Load & Ongoing Replication.
✅ Verified Data Consistency Post-Migration.

📌 Next Step: Phase 3 - Application Migration

🔗 GitHub Repo: AWS Migration Project
🎥 Watch on YouTube: Click Here

🚀 Stay tuned for more AWS migration insights!

#AWS #DMS #DatabaseMigration #CloudComputing #DevOps #RDS #AmazonWebServices

The Application Discovery Service (ADS) is the first step in migrating your workloads to AWS. It helps analyze your on-premises environment, identifying dependencies, resource usage, and configurations to streamline your migration journey.

Cloud migration is a strategic decision that requires proper planning, cost analysis, and infrastructure assessment before execution. This blog serves as a step-by-step guide to migrating workloads from an on-premises environment to AWS using Application Discovery Service (ADS).

📌 Follow the Detailed Documentation on GitHub

For in-depth step-by-step guidance on AWS migration using Application Discovery Service (ADS) and TCO analysis, refer to the full project documentation on GitHub.

🔗 GitHub Repository: AWS Migration Project

🔗 Watch the Project Video on YouTube

📄 Detailed Guides Include:
✅ Phase 1: AWS Application Discovery & TCO Analysis 📊
✅ Phase 2: Migration Planning & Execution 🚀
✅ Phase 3: Post-Migration Optimization & Scaling 📈

🌍 What is AWS Application Discovery Service (ADS)?

ADS is the first step in migrating workloads to AWS. It helps analyze on-premises infrastructure, identifying:
✅ Resource utilization (CPU, memory, storage, and networking).
✅ Application dependencies to ensure smooth transition.
✅ Right-sizing AWS infrastructure for cost optimization.

📌 Key Focus Areas of This Blog

1️⃣ Phase 1: Discovery & TCO Analysis

• Analyze on-premises workloads using ADS.

• Evaluate cost savings through a Total Cost of Ownership (TCO) analysis.

• Plan AWS infrastructure (EC2, RDS, Auto Scaling, Storage).

🌐 On-Premises Infrastructure Setup for AWS Migration

Before migrating workloads to AWS, it is crucial to understand the existing on-premises environment. This helps in determining infrastructure dependencies, resource utilization, and compatibility with AWS services. 🚀

🏢 On-Premises Setup Overview

🔹 Web Application Server

• Hosted on a local server running Apache or NGINX.

• Handles HTTP requests and serves dynamic/static content.

• Connected to the database server for backend processing.

🔹 Database Server

• Uses MySQL as the primary database.

• Stores application data, user information, and business transactions.

• Communicates with the web application over a local network.

🎉 Web Application UX – Online Banquet Booking System

A seamless user experience (UX) is the backbone of any successful online banquet booking system. Whether users are reserving banquet halls for weddings, corporate events, or private parties, the platform should be intuitive, responsive, and hassle-free.

Let’s explore the ideal UX flow for an Online Banquet Booking System and how it ensures a smooth journey for users. 🚀

🖥️ Key Features for an Intuitive UX

1️⃣ Home Page – First Impressions Matter! 🏡

• A visually appealing landing page with a search bar for quick venue lookup.

• High-quality images & videos showcasing banquet halls.

• Featured halls with ratings, price range, and capacity.

• Call-to-Action (CTA): “Find Your Perfect Venue” 🔎

2️⃣ Venue Search & Filtering 🔍

• Smart search with filters for:
  ✅ Location 📍
  ✅ Date & Availability 📅
  ✅ Capacity 🏢
  ✅ Budget 💰
  ✅ Amenities (WiFi, Parking, Catering) 🍽️

3️⃣ Venue Details Page 🏢

• Detailed descriptions of banquet halls with images, pricing, and amenities.

• Customer reviews & ratings ⭐⭐⭐⭐⭐

• Availability calendar to check open slots.

• A "Book Now" button for seamless reservations.

4️⃣ Seamless Booking Flow 🛒

• Step 1: User selects a date & time slot.

• Step 2: Chooses add-ons (catering, decoration, music, etc.) 🎶

• Step 3: Provides personal details & special requests.

• Step 4: Online payment integration 💳 (Stripe, PayPal, UPI).

• Step 5: Confirmation email & SMS 📩.

5️⃣ User Dashboard & Booking Management 🛠️

• Users can view, modify, or cancel bookings.

• Download invoices & event details 📜.

• Track upcoming reservations.

• Loyalty rewards & offers for repeat customers. 🎁

6️⃣ Admin Panel for Venue Owners 🏢

• Manage venue listings, pricing, and availability.

• Track customer bookings & payments.

• Generate reports on revenue & performance 📊.

🔍 Challenges with On-Premises Infrastructure

❌ High Maintenance Costs – Requires constant hardware upgrades and monitoring.
❌ Scalability Issues – Limited ability to scale resources dynamically.
❌ Security Risks – On-premises setups often require manual security configurations.
❌ Disaster Recovery Concerns – Risk of data loss without cloud backups or redundancy.

🔜 Next Steps: Migrating to AWS EC2 & RDS

With the AWS Application Discovery data analyzed, the next step is to migrate workloads to AWS efficiently. In Phase 2, we will:

✅ Choose the right EC2 instance types & storage
✅ Set up Auto Scaling & Load Balancers
✅ Optimize database migration with RDS or Aurora

💡 Why Migrate to AWS?

✅ Elastic Compute Scaling – Easily scale web servers using EC2 Auto Scaling.
✅ Managed Databases – Leverage Amazon RDS for automated backups and maintenance.
✅ High Availability – Deploy across multiple AWS Regions & Availability Zones.
✅ Lower TCO (Total Cost of Ownership) – Reduce upfront infrastructure costs.

🏗 What is AWS Application Discovery Service?

AWS Application Discovery Service (ADS) helps enterprises automate the collection of on-premises system metadata before migrating workloads to AWS.

How AWS ADS Works

🔹 Scans on-premises servers and collects CPU, memory, storage, and network data.
🔹 Identifies application dependencies and usage patterns.
🔹 Provides insights for right-sizing AWS infrastructure post-migration.

AWS ADS supports two discovery methods:
1️⃣ Agent-Based Discovery (For deep-level data collection, including system processes).
2️⃣ Agentless Discovery (For VMware-based environments).

➡️ Detailed Documentation: AWS ADS Official Guide

🎯 Why Use AWS ADS for Migration?

✅ Comprehensive Data Collection: Automates server discovery and provides real-time metrics.
✅ Migration Planning: Helps identify dependencies and application workloads.
✅ Optimized AWS Resources: Right-size EC2 and RDS instances post-migration.
✅ Cost-Saving: Eliminates over-provisioning by analyzing actual resource usage.

⚙️ Pre-Requisites & Deployment

Before deploying AWS ADS, ensure the following:

🔹 1️⃣ Pre-Requisite Setup

✅ Configure IAM Roles & Policies for ADS to access on-premises servers.
✅ Verify network connectivity between your on-premises servers and AWS.
✅ Ensure your firewall rules allow ADS communication.

📄 Step-by-Step Guide: Pre-Requisite Setup

🔹 2️⃣ Deploy AWS ADS

🚀 Steps to deploy:
1️⃣ Install AWS Discovery Agent on on-premises servers.
2️⃣ Configure agent-based or agentless discovery (depending on your infrastructure).
3️⃣ Set up AWS Migration Hub to monitor discovery insights.
4️⃣ Verify ADS is collecting data from your on-premises workloads.

📄 Detailed Guide: AWS ADS Deployment

🎯 Key Benefits of AWS ADS

✅ 1. Data-Driven Migration Decisions

✔️ Collect real-time data on CPU, memory, and disk utilization.
✔️ Avoid assumptions and right-size AWS instances based on actual resource usage.

🔗 2. Application Dependency Mapping

✔️ Identify interconnected workloads that must be migrated together.
✔️ Reduce post-migration downtime and compatibility issues.

💰 3. Cost Optimization

✔️ Prevent over-provisioning AWS resources.
✔️ Ensure efficient workload placement based on real-world usage metrics.

📊 4. Faster Migration Planning

✔️ Reduce planning time using automated discovery insights.
✔️ Eliminate the need for manual infrastructure assessments.

🚀 Phase 1: AWS Application Discovery & TCO Analysis

Cloud migration is more than just moving workloads to the cloud. Understanding your existing infrastructure, dependencies, and costs is critical before making the move. In Phase 1, we leveraged AWS Application Discovery Service (ADS) to analyze our environment and performed a Total Cost of Ownership (TCO) Analysis to estimate the potential savings and efficiency improvements.

This blog walks you through the key steps, tools used, and what we achieved in this phase! 📊

📌 Key Steps in Phase 1

🔹 AWS Application Discovery & TCO Analysis

• 📄 Pre-Requisite: Setup Requirements

• 📄 Discovery Service Deployment: Collecting Data

• 📄 High-Level TCO Analysis: Cost Breakdown

• 📄 Complete AWS Migration Documentation

🔍 What We Achieved in Phase 1

✅ 1. Setting Up AWS Migration Prerequisites

📄 Discovery Service Deployment: Collecting Data

• Configured AWS Migration Hub to track migration progress 🌍.

• Created necessary IAM roles and permissions for secure access 🔐.

• Defined AWS Application Discovery Service (ADS) setup to analyze infrastructure.

🛠 2. Deploying AWS Discovery Service

• Installed ADS Agents on Web and Database servers to collect system data 🖥️📊.

Data Collectors:

WebServer

Technical Information

Performance Information

DbServer

Technical Information

Performance Information

• Verified network dependencies, CPU utilization, memory usage, and active processes.

WebServer

DbServer

• Organized servers into Application Groups for better classification.

💰 3. High-Level TCO (Total Cost of Ownership) Analysis

• Compared on-premise costs vs. AWS projected costs across:

  • Compute (EC2 vs. on-prem servers) ⚙️

  • Storage (S3, EBS vs. local storage) 📦

  • Networking (AWS Data Transfer vs. existing network costs) 🌐

  • Database (AWS RDS vs. on-prem SQL/Oracle) 🗄️

• Identified potential cost savings and ROI for cloud migration 📉.

🛠️ Tools Used in This Phase

🔹 AWS Application Discovery Service (ADS) – Collected system performance, network dependencies, and server data.
🔹 AWS Migration Hub – Centralized dashboard for tracking server inventory and migration status.
🔹 IAM (Identity and Access Management) – Created roles and permissions for secure access.
🔹 CloudTrail – Monitored activities and logs related to migration actions.
🔹 Application Groups – Used in Migration Hub to organize and manage servers efficiently.

📊 High-Level TCO Analysis for WebServer & DatabaseServer Migration to AWS

📋 Overview

As organizations migrate their Web Servers and Database Servers from on-premises infrastructure to AWS Cloud, performing a Total Cost of Ownership (TCO) analysis is crucial. This document evaluates the on-prem vs AWS cloud cost breakdown, helping decision-makers optimize costs, improve scalability, and enhance security.

✅ What This Guide Covers

• 📊 TCO breakdown of on-prem vs. AWS

• 💻 Recommended AWS services for Web & Database servers

• 💰 Projected cost savings and optimization strategies

For a detailed documentation on the TCO (Total Cost of Ownership) Analysis, visit:

📄 📌 High-Level TCO Analysis Documentation

📌 Current On-Premises Infrastructure Overview

📌 Web Server Analysis (ip-10-0-1-82) 💻

🔹 Application: WebApp
🔹 OS: Ubuntu 24.04.1 LTS
🔹 CPU: 1 vCPU (x86_64)
🔹 RAM: 1 GB
🔹 Storage: 8 GB SSD
🔹 Hypervisor: Xen
🔹 Network Interfaces: 1

Performance Metrics 📊

• CPU Usage: 0.35% 🖥️

• RAM Usage: 38.77% 💾

• Disk Reads: 1.59 KBPS 📥

• Disk Writes: 2.82 KBPS 📤

💡 Observations:
✅ CPU usage is minimal, meaning we don’t need a high-end instance.
✅ RAM consumption is moderate (38.77%), but might need an upgrade in production.
✅ Disk I/O is low, so an EBS gp3 volume would be a good fit post-migration.

🗄️ Database Server Analysis (ip-10-0-2-254)

🔹 Application: DatabaseServer
🔹 OS: Ubuntu 24.04.1 LTS
🔹 CPU: 1 vCPU (x86_64)
🔹 RAM: 1 GB
🔹 Storage: 8 GB SSD
🔹 Hypervisor: Xen
🔹 Network Interfaces: 1

Performance Metrics 📊

• CPU Usage: 0.31% 🖥️

• RAM Usage: 43.34% 💾

• Disk Reads: 0.86 KBPS 📥

• Disk Writes: 1.79 KBPS 📤

💡 Observations:
✅ Low CPU utilization suggests no need for a compute-heavy instance.
✅ Higher RAM usage (43.34%) indicates potential memory constraints.
✅ Disk I/O is light, making gp3 SSD storage an optimal choice.

🛠️ Migration Considerations & Recommendations

Based on this performance analysis, here’s our recommended AWS migration plan:

🔹 EC2 Instance Sizing:

• Web Server → t3.micro (2 vCPU, 1GB RAM)

• Database Server → t3.small (2 vCPU, 2GB RAM) (Upgrade for better performance)

🔹 Storage Optimization:

• Use gp3 EBS volumes for both servers to balance cost and performance.

🔹 Auto Scaling:

• Configure AWS Auto Scaling to handle unexpected traffic spikes.

🔹 Database Optimization:

• Consider Amazon RDS for managed PostgreSQL/MySQL instead of self-managed DB.

📊 TCO Breakdown: On-Premises vs. AWS

💰 On-Premises Cost Estimate

☁️ AWS Cost Estimate (EC2 + RDS)

📉 TCO Savings Analysis

✅ AWS Migration reduces costs by ~85%
✅ Significant reduction in maintenance, networking, and power costs
✅ AWS RDS provides auto-scaling & automatic failover for databases

📌 Next Steps - How to Perform a Detailed TCO Analysis

Step 1: Identify Current Costs

📌 Analyze existing hardware, software, networking, maintenance, security costs.
📌 Calculate power, cooling, IT support, disaster recovery expenses.

Step 2: Map AWS Services

📌 Identify right-sized EC2 instances, Amazon RDS, storage options.
📌 Estimate AWS Networking, Backup, and Security Costs.

Step 3: Calculate AWS TCO

📌 Use AWS Pricing Calculator → AWS TCO Calculator
📌 Compare on-premises annual costs vs AWS annual costs.

Step 4: Optimize AWS Costs

📌 Consider Savings Plans, Reserved Instances, or Spot Instances for long-term savings.
📌 Implement Auto-Scaling & Monitoring to avoid over-provisioning.

🎯 Conclusion

Migrating Web Servers & Database Servers to AWS provides:
✅ 85% cost savings compared to on-prem infrastructure
✅ Scalability & high availability via AWS Auto-Scaling, RDS Multi-AZ
✅ Reduced IT maintenance using managed AWS services
✅ Enhanced security & monitoring with AWS IAM, WAF, and CloudWatch

🚀 Ready to migrate? Start by running AWS Application Discovery Service (ADS) for automatic migration insights.

📖 References

• AWS TCO Calculator

• AWS EC2 Pricing

• AWS Migration Hub

• AWS Well-Architected Framework

  📌 AWS Application Discovery Service (ADS)
  📌 AWS Database Migration Service (DMS)
  📌 AWS Migration Hub

Phase 2 - Database Migration & Execution

With Phase 1 complete, we now have full visibility into our infrastructure, dependencies, and costs. The next step is migration planning and execution!

📌 In Phase 2, we will:
✅ Choose the right AWS services for hosting applications
✅ Develop an automated infrastructure strategy (Terraform, CloudFormation)
✅ Optimize our AWS cost management for long-term savings

💬 Have questions about AWS migration? Drop a comment below!
📢 Follow for more real-world AWS Cloud Migration insights! 🚀

💡 Why This Blog?

This guide is designed for IT professionals, cloud architects, and DevOps engineers who want to:
✅ Understand AWS migration best practices.
✅ Perform detailed cost comparisons between on-prem and AWS.
✅ Use AWS services effectively to ensure a smooth migration.

Whether you’re planning a small-scale migration or a large enterprise transition, this blog will help you navigate each phase efficiently!

📢 Stay tuned for real-world AWS migration insights and hands-on tutorials! 🚀

🔗 GitHub Repository: mgn-aws-project01

🧑‍💻 Author

👨‍💻 Created and maintained by Praful Patel.
🔗 GitHub | 🌍 Tech Blog

This project showcases a fully serverless video upload and playback application built using AWS services. The application enables users to upload videos, store them in Amazon S3, and manage metadata in DynamoDB. Videos can then be fetched and played seamlessly through a modern web interface. The architecture is scalable, secure, and cost-effective.

👉 Follow Project on GitHub Repo Link: https://github.com/prafulpatel16/video-app-aws-serverless/blob/master/README.md 🚀✨

Architecture Diagram

Key Features

1. Scalable serverless architecture.

2. Upload videos directly from the frontend to S3.

3. Store video metadata in DynamoDB.

4. Fetch and play videos via a modern web interface.

Tech Stack

• Frontend: HTML, CSS, JavaScript

• Backend: AWS Lambda, API Gateway

• Database: DynamoDB

• Storage: S3

• Monitoring: CloudWatch

Step-by-Step Implementation

1. Setting Up Amazon S3 for Video Storage

1. Create an S3 Bucket:

    aws s3 mb s3://video-upload-bucket
   

1. Configure CORS:

    [
        {
            "AllowedHeaders": ["*"],
            "AllowedMethods": ["GET", "PUT", "POST"],
            "AllowedOrigins": ["*"]
        }
    ]
   

2. Enable Static Website Hosting:

   • Go to the Properties tab in the S3 console.

   • Set Index Document to index.html.

1. Sync Frontend Files:

    aws s3 sync static-web/ s3://video-upload-bucket
   

2. Setting Up DynamoDB for Metadata Storage

1. Create a DynamoDB Table:

    aws dynamodb create-table \
        --table-name video-metadata \
        --attribute-definitions AttributeName=videoId,AttributeType=S \
        --key-schema AttributeName=videoId,KeyType=HASH \
        --billing-mode PAY_PER_REQUEST
   

   ---

3. Creating Lambda Functions

Upload Handler

This function uploads videos to S3 and saves metadata in DynamoDB.

import boto3
import json
import os
import uuid

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')

BUCKET_NAME = os.environ['BUCKET_NAME']
TABLE_NAME = os.environ['TABLE_NAME']

def lambda_handler(event, context):
    try:
        file_content = event['body']
        file_name = f"{uuid.uuid4()}.mp4"

        # Upload video to S3
        s3.put_object(Bucket=BUCKET_NAME, Key=file_name, Body=file_content)

        # Save metadata to DynamoDB
        table = dynamodb.Table(TABLE_NAME)
        table.put_item(
            Item={
                'videoId': file_name,
                'url': f"https://{BUCKET_NAME}.s3.amazonaws.com/{file_name}"
            }
        )

        return {
            "statusCode": 200,
            "headers": {"Access-Control-Allow-Origin": "*"},
            "body": json.dumps({"message": "File uploaded successfully"})
        }
    except Exception as e:
        return {
            "statusCode": 500,
            "headers": {"Access-Control-Allow-Origin": "*"},
            "body": json.dumps({"error": str(e)})
        }

Fetch Handler

This function retrieves video metadata from DynamoDB.

import boto3
import json
import os

dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['TABLE_NAME']

def lambda_handler(event, context):
    try:
        table = dynamodb.Table(TABLE_NAME)
        response = table.scan()

        return {
            "statusCode": 200,
            "headers": {"Access-Control-Allow-Origin": "*"},
            "body": json.dumps(response['Items'])
        }
    except Exception as e:
        return {
            "statusCode": 500,
            "headers": {"Access-Control-Allow-Origin": "*"},
            "body": json.dumps({"error": str(e)})
        }

4. Configuring API Gateway

1. Create an API:

   • Go to API Gateway > Create API.

   • Choose HTTP API and name it video-app-api.

2. Add Routes:

   • POST /upload → video-upload-handler.

   • GET /fetch → video-fetch-handler.

3. Enable CORS:

   • Add headers:

     • Access-Control-Allow-Origin: *

     • Access-Control-Allow-Methods: GET, POST, OPTIONS

5. Hosting Frontend on S3

1. Sync the frontend files:

    aws s3 sync static-web/ s3://video-upload-bucket
   

6. Testing the Application

1. Open the static website URL in your browser.

2. Upload videos and fetch metadata to test the functionality.

Monitoring and Optimization

1. Enable CloudWatch Logs for Lambda functions to monitor errors and performance.

2. Use CloudFront to distribute video content globally for faster playback.

Challenges and Solutions

1. CORS Issues:

   • Ensure proper headers are configured in API Gateway and Lambda responses.

2. Large File Uploads:

   • Use multipart uploads for better performance with large files.

Conclusion

This AWS serverless project demonstrates the power of building scalable, cost-efficient, and secure video upload and playback systems. By leveraging AWS services like S3, DynamoDB, Lambda, and API Gateway, developers can deliver high-quality user experiences without managing server infrastructure. This architecture is ideal for real-time video applications.

GitHub Repo: https://github.com/prafulpatel16/prafuls-portfolio-webapp

Solution Diagram: AWS Serverless Architecture

Frontend:

1. HTML/CSS/JavaScript: For building the website structure and interactivity.

2. Bootstrap: For responsive design and styling.

3. Google Fonts: For custom fonts.

4. JavaScript Libraries:

   • Typed.js: For typing animation on the website.

   • PureCounter.js: For dynamic visitor counter on the frontend.

   • AOS.js: For animations on scroll.

   • Swiper.js: For testimonial slider functionality.

Backend:

1. AWS Lambda: For serverless functions that handle resume downloads and visitor counting.

2. AWS API Gateway: For exposing API endpoints for the Lambda functions.

3. AWS DynamoDB: For storing visitor counts and incrementing them for each visit.

4. AWS S3: For storing the resume PDF file that users can download.

Cloud Infrastructure & Security:

1. AWS IAM: For managing roles and policies to secure Lambda access to S3 and DynamoDB.

2. AWS CloudWatch: For logging and monitoring Lambda functions.

3. AWS WAF (optional): For protecting the API Gateway endpoints (not implemented but recommended).

Version Control & Project Management:

1. Git: For version control.

2. GitHub: For hosting and collaboration on the project.

Scripting:

1. Python: Used in Lambda functions to handle resume downloads and visitor counter logic.

2. Boto3: Python SDK to interact with AWS services (S3, DynamoDB, etc.).

Deployment & Hosting:

1. AWS Free Tier: Keeping costs within the AWS Free Tier by leveraging free-tier limits on Lambda, API Gateway, S3, and DynamoDB.

Table of Contents

1. Introduction

2. Project Structure

3. System Design Overview

4. Infrastructure Setup

   • IAM Role Setup

   • Lambda Function Setup

   • API Gateway Setup

5. Resume Download Functionality

   • Implementation Overview

   • S3 Configuration

   • API Gateway-Lambda Integration for Resume Download

6. Visitor Counter Functionality

   • Implementation Overview

   • DynamoDB Configuration

   • API Gateway-Lambda Integration for Visitor Counter

7. Security Measures

8. Budget Considerations

9. Deployment Diagram

10. Final Testing & Validation

1. Introduction

This project demonstrates how to build a serverless web application with a Resume Download Functionality and Visitor Counter using AWS services. The primary focus is on leveraging API Gateway, Lambda Functions, S3, DynamoDB, and IAM roles. This system ensures that the infrastructure operates efficiently within the AWS Free Tier limits and maintains a budget of $10/month for all AWS resources.

2. Project Structure

bashCopy code/project-root
    ├── /forms
    │   └── insert.php (for contact form)
    ├── /assets
    │   ├── /css (for stylesheets)
    │   ├── /img (for images, including the resume)
    │   └── /js (for JavaScript)
    ├── index.php (Main Website)
    └── README.md (Project Documentation)

3. System Design Overview

System Architecture

The architecture consists of:

1. Frontend: A simple HTML/Bootstrap-based web page hosted via a static site generator (e.g., AWS S3 or EC2).

2. Backend Services: Lambda functions that handle:

   • Resume Download: Fetching the resume from an S3 bucket and allowing the user to download it.

   • Visitor Counter: Logging visits in a DynamoDB table and returning the visitor count.

Core AWS Services Used:

• S3: Stores the resume PDF.

• Lambda: Handles the backend logic (fetching the resume and tracking visitor count).

• API Gateway: Exposes API endpoints for Lambda functions.

• DynamoDB: Stores visitor count.

• IAM Roles: Manages access permissions.

4. Infrastructure Setup

4.1 IAM Role Setup

Steps:

1. Create IAM Role for Lambda Execution:

   • Go to IAM in the AWS console.

   • Click on "Roles" > "Create Role."

   • Choose "Lambda" as the trusted entity.

   • Attach the following policies:

     • AmazonS3ReadOnlyAccess: Allows Lambda to read from the S3 bucket.

     • AmazonDynamoDBFullAccess: Allows Lambda to interact with DynamoDB.

     • AWSLambdaBasicExecutionRole: Grants Lambda access to CloudWatch for logs.

   • Give the role a meaningful name (e.g., Lambda-Resume-Download-Role).

1. Policy for S3 Access: Ensure the following policy is attached:

    code{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
      ]
    }
   

4.2 Lambda Function Setup

Steps to Create Lambda Function for Resume Download:

1. Create Lambda Function:

   • Go to AWS Lambda and click "Create Function."

   • Choose the "Author from scratch" option.

   • Name the function ResumeDownloadFunction.

   • Choose the runtime (Python 3.9 or Node.js as per your choice).

   • Set the IAM role to the one you created (Lambda-Resume-Download-Role).

2. Lambda Code:

    import json
    import boto3
    import base64
    import os
    from botocore.exceptions import ClientError
   
    s3 = boto3.client('s3')
   
    def lambda_handler(event, context):
        bucket_name = os.getenv('S3_BUCKET_NAME')
        resume_key = os.getenv('RESUME_KEY')
   
        try:
            # Fetch the resume PDF from S3
            response = s3.get_object(Bucket=bucket_name, Key=resume_key)
            pdf_content = response['Body'].read()
   
            # Return the PDF file as binary stream
            return {
                'statusCode': 200,
                'headers': {
                    'Content-Type': 'application/pdf',
                    'Content-Disposition': 'attachment; filename="Praful_Resume.pdf"',
                },
                'body': base64.b64encode(pdf_content).decode('utf-8'),
                'isBase64Encoded': True  # This must be True for binary files
            }
   
        except ClientError as e:
            error_code = e.response['Error']['Code']
            return {
                'statusCode': 500,
                'body': json.dumps(f"Error downloading resume: {error_code} - {str(e)}")
            }
        except Exception as e:
            return {
                'statusCode': 500,
                'body': json.dumps(f"Error downloading resume: {str(e)}")
            }
   

3. Set Environment Variables:

   • S3_BUCKET_NAME: Your S3 bucket name.

   • RESUME_KEY: The key (path) to your resume PDF file in S3.

Steps to Create Lambda Function for Visitor Counter:

1. Create a second Lambda function called VisitorCounterFunction using the same steps as above.

2. Lambda Code:

    import json
    import boto3
    from decimal import Decimal
   
    # Initialize DynamoDB resource
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('visitorCounterTable')
   
    # Helper function to convert Decimal to int or float
    def decimal_default(obj):
        if isinstance(obj, Decimal):
            return int(obj) if obj % 1 == 0 else float(obj)
        raise TypeError
   
    def lambda_handler(event, context):
        try:
            # Define the primary key (static 'id' for counting visitors)
            visitor_id = "visitorCount"
   
            # Update the visitor count in DynamoDB
            response = table.update_item(
                Key={'id': visitor_id},
                UpdateExpression="SET visits = if_not_exists(visits, :start) + :increment",
                ExpressionAttributeValues={
                    ':start': 0,
                    ':increment': 1
                },
                ReturnValues="UPDATED_NEW"
            )
   
            # Get the updated visitor count
            updated_visits = response['Attributes']['visits']
   
            # Return response with CORS headers
            return {
                'statusCode': 200,
                'headers': {
                    'Access-Control-Allow-Origin': '*',  # Allow any origin
                    'Access-Control-Allow-Methods': 'GET',  # Allow GET method
                    'Access-Control-Allow-Headers': 'Content-Type',  # Allow Content-Type header
                },
                'body': json.dumps({'visits': updated_visits}, default=decimal_default)
            }
   
        except Exception as e:
            print(f"Error updating visitor count: {str(e)}")
            return {
                'statusCode': 500,
                'headers': {
                    'Access-Control-Allow-Origin': '*',
                    'Access-Control-Allow-Methods': 'GET',
                    'Access-Control-Allow-Headers': 'Content-Type',
                },
                'body': json.dumps({'message': f'Error: {str(e)}'})
            }
   

4.3 API Gateway Setup

Steps:

1. Create a new API in API Gateway.

   • Go to API Gateway in the AWS Console.

   • Create a REST API.

   • Name it resumeDownload

2. Create Resource and Method for Resume Download:

   • Create a resource /resume.

   • Under /resume, create a GET method.

   • Integrate the method with the ResumeDownloadFunction Lambda function.

1. Create Resource and Method for Visitor Counter:

   • Create a resource /visitCount.

   • Under /visitCount, create a GET method.

   • Integrate the method with the VisitorCounterFunction Lambda function.

2. Enable CORS for both endpoints in API Gateway.

3. Deploy API:

   • Go to "Deploy API" in API Gateway.

   • Create a new stage (e.g., dev).

   • Note down the endpoint URL for both /resume and /visitCount.

5. Resume Download Functionality

5.1 Implementation Overview

• The resume is stored as a PDF in an S3 bucket.

• The Lambda function fetches the resume from S3 and sends it to the user.

• The API Gateway acts as the entry point, invoking the Lambda function.

5.2 S3 Configuration

1. Create an S3 Bucket:

   • Go to S3 in the AWS Console.

   • Create a bucket (e.g., resume-bucket).

   • Upload your resume as a PDF file.

   • Ensure the file is accessible via the S3 read permissions set in the IAM role.

5.3 API Gateway-Lambda Integration

• The /resume endpoint calls the Lambda function to fetch the resume.

• The Lambda function fetches the resume from S3 and encodes it in base64 format to be sent to the user via API Gateway.

6. Visitor Counter Functionality

6.1 Implementation Overview

• A DynamoDB table stores the visitor count.

• The Lambda function increments the count each time it is invoked and returns the updated count.

6.2 DynamoDB Configuration

1. Create a DynamoDB Table:

   • Go to DynamoDB in the AWS Console.

   • Create a table called VisitorCounter.

   • Set id as the partition key.

   • Prepopulate the table with an item:

       jsonCopy code{
         "id": "visitor_count",
         "visit_count": 0
       }
     

API Deployed

6.3 API Gateway-Lambda Integration

• The /visitCount endpoint calls the Lambda function, which increments and returns the updated visitor count.

Access web application through S3 static webapp

7. Security Measures

• IAM Role: Ensure that the Lambda function has only the required permissions (S3 read, DynamoDB access).

• API Gateway Authorization: Consider adding an API key or Cognito for access control.

• S3 Bucket: Enable server-side encryption (SSE-S3 or SSE-KMS) for the resume file.

• AWS WAF: Add a Web Application Firewall (WAF) to protect your API endpoints.

8. Budget Considerations

• Lambda: AWS Lambda comes with 1M free requests and 400,000 GB-seconds of compute time, which should be sufficient for low-traffic websites.

• S3: S3 provides 5GB of free storage and 20,000 GET requests/month within the Free Tier.

• DynamoDB: DynamoDB offers 25 RCU/WCU (Read/Write Capacity Units) and 25GB of free storage per month.

• API Gateway: The Free Tier includes 1M REST API calls per month.

By staying within these limits, the overall cost will remain under $10.

9. Deployment Diagram

10. Final Testing & Validation

1. Test API Endpoints:

   • Ensure the /resume endpoint returns the PDF correctly.

   • Ensure the /visitCount endpoint increments and returns the visitor count.

2. Frontend Integration:

   • Integrate the API URLs with the frontend (index.php).

   • Test the resume download and visitor counter.

3. Monitor Logs:

   • Use CloudWatch to monitor Lambda executions and catch any potential errors.

This documentation provides a comprehensive guide to setting up a serverless web application using AWS services like Lambda, API Gateway, S3, and DynamoDB. Follow these steps closely to implement the resume download and visitor counter functionalities within a budget of $10.

GitHub Repo Link: https://github.com/prafulpatel16/aws-order-proccessing-system.git

AWS Serverless offerings

Project Use Case: Real-Time Order Processing System

Architecture Overview:

• User Interface (UI): A React frontend hosted on S3 and served via CloudFront.

• Backend: API Gateway, AWS Lambda functions, Step Functions for order processing orchestration, and DynamoDB as the database.

• Additional Services: SNS for notifications, S3 for storing receipts, and CloudWatch for monitoring.

1. Frontend (React): A simple order form hosted in an S3 bucket.

2. API Gateway: To handle order submission requests.

3. Lambda: Multiple Lambda functions for each stage of the order processing.

4. Step Functions: Orchestration for the order processing workflow.

5. DynamoDB: For storing orders and inventory data.

6. SQS: Queue to process background tasks like sending notifications and generating receipts.

7. SNS: For real-time notifications to users.

8. S3: For storing order receipts.

9. CloudWatch: For monitoring and error logging.

Key Requirements:

1. Real-time Order Submission: Users can place orders through the e-commerce frontend.

2. Order Validation: Validate the order, including checking stock availability and payment verification.

3. Inventory Management: Deduct inventory once the order is placed.

4. Payment Processing: Integrate with third-party payment gateways.

5. Notification: Notify users via email when the order is successfully processed.

6. Store Order Receipts: Store the order details and generate a receipt to be stored in S3.

7. Monitoring: Use CloudWatch to monitor the flow, errors, and execution times.

Tech Stack:

• Frontend: React.js (Hosted in S3 + CloudFront)

• API: API Gateway (to expose REST API)

• Logic: Lambda functions

• Orchestration: Step Functions

• Database: DynamoDB (for order details and inventory management)

• Notifications: SNS

• File Storage: S3 (for storing receipts)

• Monitoring: CloudWatch

Step-by-Step Implementation:

1. Frontend (React + API Gateway):

• Create a React application for order submission.

• Host the React frontend in S3 with CloudFront for faster access.

• The frontend sends an API request to the API Gateway to submit the order.

• API Gateway triggers a Lambda function to start the process.

2. API Gateway Setup:

• Configure AWS API Gateway to expose a REST API with a /place-order endpoint.

• This API will trigger an AWS Lambda function (OrderPlacementFunction).

• The Lambda function will initiate an AWS Step Functions workflow.

3. AWS Step Functions:

• Define a Step Function to manage the order processing workflow.

• The workflow consists of multiple states:

  • Validate Order: Check for stock availability using Lambda.

  • Process Payment: Trigger payment processing using a Lambda function.

  • Update Inventory: Once payment is successful, deduct the inventory.

  • Send Notification: Send a confirmation email via SNS.

  • Generate Receipt: Store the order receipt in S3 using Lambda.

4. Order Validation Lambda:

• Create a Lambda function (ValidateOrderFunction) that validates the stock availability by querying DynamoDB.

• If the item is in stock, the workflow proceeds to payment processing.

5. Payment Processing Lambda:

• Lambda function (ProcessPaymentFunction) integrates with a third-party payment service (e.g., Stripe).

• After successful payment, update the payment status in DynamoDB.

6. Update Inventory Lambda:

• Lambda function (UpdateInventoryFunction) updates the inventory in DynamoDB once the payment is processed.

• If inventory update fails, trigger a rollback or handle errors via a defined Step Functions fail state.

7. Send Notification (SNS):

• Create an SNS topic to send a notification to the user about the order status.

• Lambda function (SendNotificationFunction) triggers SNS to send an email with the order details to the user.

8. Generate and Store Receipt (S3):

• Lambda function (GenerateReceiptFunction) generates a receipt for the order and stores it in an S3 bucket.

• A presigned URL is generated for users to download the receipt.

9. Monitoring and Error Handling:

• Use AWS CloudWatch to track the workflow and log errors.

• Step Functions should have proper error handling with retry logic or defined failure states.

• CloudWatch metrics and alarms can be set to monitor for errors in the order process.

AWS Step Functions Workflow Example:

Copy{
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ValidateOrderFunction",
      "Next": "ProcessPayment",
      "Catch": [
        {
          "ErrorEquals": ["States.TaskFailed"],
          "Next": "FailOrder"
        }
      ]
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ProcessPaymentFunction",
      "Next": "UpdateInventory"
    },
    "UpdateInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:UpdateInventoryFunction",
      "Next": "SendNotification"
    },
    "SendNotification": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:SendNotificationFunction",
      "Next": "GenerateReceipt"
    },
    "GenerateReceipt": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:GenerateReceiptFunction",
      "End": true
    },
    "FailOrder": {
      "Type": "Fail",
      "Error": "OrderProcessingFailed",
      "Cause": "An error occurred during order processing."
    }
  }
}

Benefits of this Approach:

• Scalable and Serverless: No server management needed, the architecture scales automatically with the load.

• Event-Driven: AWS Step Functions allow orchestration of Lambda functions in a step-by-step fashion.

• Real-Time Notifications: SNS ensures users are notified instantly once the order is processed.

• Cost-Effective: You only pay for the compute resources (Lambda executions) and API Gateway usage.

• Monitoring: CloudWatch allows monitoring in real time for better insight into performance and errors.Tech Stack:

  • Frontend: React.js (Hosted in S3 + CloudFront)

  • API: API Gateway (to expose REST API)

  • Logic: Lambda functions

  • Orchestration: Step Functions

  • Database: DynamoDB (for order details and inventory management)

  • Notifications: SNS

  • File Storage: S3 (for storing receipts)

  • Monitoring: CloudWatch

Project Structure:

Copyorder-processing-system/
├── frontend/                   # React app for frontend
│   ├── public/
│   ├── src/
│   └── package.json
├── backend/
│   ├── functions/              # Lambda functions
│   │   ├── validateOrder.js
│   │   ├── processPayment.js
│   │   ├── updateInventory.js
│   │   ├── sendNotification.js
│   │   ├── generateReceipt.js
│   └── stepFunctions.json      # Step Function definition
├── infrastructure/             # Infrastructure as Code (CloudFormation/Terraform)
│   ├── api-gateway.yaml
│   ├── dynamodb.yaml
│   ├── s3.yaml
│   ├── sqs.yaml
│   ├── sns.yaml
│   └── step-functions.yaml
└── README.md                   # Project documentation

Project Implementation:

Dynamodb Setup

To launch an individual CloudFormation stack using a CloudFormation template (in your case, dynamodb.yaml) that has been uploaded to S3, follow the steps below:

Step 1: Upload the CloudFormation Template to S3

You mentioned you have already uploaded the dynamodb.yaml file to S3. Here's the command you would use if you haven't:

aws s3 cp dynamodb.yaml s3://your-bucket-name/
Replace your-bucket-name with the name of your actual S3 bucket.

Step 2: Launch the CloudFormation Stack Using the Template

You can launch the CloudFormation stack using the AWS CLI by referencing the template in S3.

Here’s how to launch the stack:

1. Run the following command to create a CloudFormation stack:

aws cloudformation create-stack \
  --stack-name dynamodb-stack \
  --template-url https://s3.amazonaws.com/your-bucket-name/dynamodb.yaml \
  --capabilities CAPABILITY_NAMED_IAM
Replace your-bucket-name with your S3 bucket name.

• Replace dynamodb.yaml with the path to the CloudFormation template.

• The --capabilities CAPABILITY_NAMED_IAM flag allows CloudFormation to create IAM roles or policies if necessary.

Explanation of Command:

• --stack-name dynamodb-stack: The name you want to give the CloudFormation stack. This can be anything meaningful, such as dynamodb-stack.

• --template-url: The URL of the CloudFormation template stored in your S3 bucket.

• --capabilities CAPABILITY_NAMED_IAM: This flag is required if your CloudFormation stack creates IAM resources such as roles or policies.

Step 3: Monitor the Stack Creation

Once the command is executed, you can monitor the progress of your stack creation either through the AWS Management Console or using the CLI.

• To check the status of the stack using the CLI, you can run:

aws cloudformation describe-stacks --stack-name dynamodb-stack
This will give you details about the status of the stack (e.g., CREATE_IN_PROGRESS, CREATE_COMPLETE, etc.).

Step 4: Verify Stack Creation

1. AWS Management Console:

   • Go to the CloudFormation section in the AWS Management Console.

   • Find your stack (dynamodb-stack) in the list of stacks and check its status.

   • You can also look at the Resources tab to see the DynamoDB table and other resources created by your stack.

2. AWS CLI:

   • You can describe the resources created by your stack using this command:

    aws cloudformation describe-stack-resources --stack-name dynamodb-stack
    This will list the resources (e.g., DynamoDB tables) that have been created by the stack.

Step 5: Interact with the Created Resources

Once your stack is successfully created, you can interact with the resources (such as the DynamoDB table) using the AWS Management Console or the AWS CLI.

For example, to list the tables in DynamoDB:

aws dynamodb list-tables

Inventory Table Schema:

1. Partition Key (Primary Key):

   • productId: A unique identifier for each product (String type).

2. Attributes:

   • stock: The available stock quantity for the product (Number type).

   • price: The price of the product (optional, Number type).

   • description: A description of the product (optional, String type).

Table Definition:

Example Table Definition (AWS CLI):

You can create the table using the AWS CLI as follows:

aws dynamodb create-table \
    --table-name Inventory \
    --attribute-definitions AttributeName=productId,AttributeType=S \
    --key-schema AttributeName=productId,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Example Data:

Here is some sample data for the Inventory table:

Adding Sample Data (AWS CLI):

You can add items to your DynamoDB table using the AWS CLI:

aws dynamodb put-item \
    --table-name Inventory \
    --item '{
        "productId": {"S": "P001"},
        "stock": {"N": "100"},
        "price": {"N": "19.99"},
        "description": {"S": "Red T-shirt"}
    }'

aws dynamodb put-item \
    --table-name Inventory \
    --item '{
        "productId": {"S": "P002"},
        "stock": {"N": "50"},
        "price": {"N": "29.99"},
        "description": {"S": "Blue Jeans"}
    }'

Verifying Data:

To verify the data inserted into your DynamoDB table, you can use the following command:

aws dynamodb scan --table-name Inventory

Example Output:

code{
    "Items": [
        {
            "productId": { "S": "P001" },
            "stock": { "N": "100" },
            "price": { "N": "19.99" },
            "description": { "S": "Red T-shirt" }
        },
        {
            "productId": { "S": "P002" },
            "stock": { "N": "50" },
            "price": { "N": "29.99" },
            "description": { "S": "Blue Jeans" }
        }
    ],
    "Count": 2,
    "ScannedCount": 2
}

Conclusion:

• Partition Key: Use productId as the partition key, which will uniquely identify each product.

• Attributes: Store the available stock as a number, and optionally add price and description attributes for each product.

INSERT some items in Inventory data table as stock so it can Validate the order

aws dynamodb put-item \ --table-name Inventory \ --item '{ "productId": {"S": "P001"}, "stock": {"N": "100"}, "price": {"N": "19.99"}, "description": {"S": "Red T-shirt"} }'

Lambda function

Phase 1: Deploy API Gateway to Trigger Lambda

Step 1.1: Create a Lambda Function (OrderPlacementFunction)

1. Go to the Lambda Console.

2. Click Create Function.

   • Function Name: OrderPlacementFunction

   • Runtime: Node.js 14.x (or the runtime of your choice)

   • Role: Choose or create an IAM role that allows Lambda to interact with AWS Step Functions.

3. In the Lambda code editor, add code that initiates the AWS Step Functions workflow when an order is placed:

orderPlacement.js (Lambda code):

const WS = require('aws-sdk');
const stepFunctions = new AWS.StepFunctions();

exports.handler = async (event) => {
  const order = JSON.parse(event.body);  // Assuming the order details are in the body

  const params = {
    stateMachineArn: process.env.STEP_FUNCTION_ARN,  // ARN of the Step Functions state machine
    input: JSON.stringify(order),  // Pass the order details to Step Functions
  };

  try {
    const result = await stepFunctions.startExecution(params).promise();
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: 'Order processing started',
        executionArn: result.executionArn,
      }),
    };
  } catch (error) {
    console.error(error);
    return {
      statusCode: 500,
      body: JSON.stringify({ message: 'Failed to start order processing' }),
    };
  }
};

4. Environment Variable:

   • Add an environment variable to store the Step Functions ARN (STEP_FUNCTION_ARN).

   • The value will be set later once the Step Functions workflow is created.

5. Deploy the Lambda Function.

ENV variable

arn:aws:iam::202533534284:role/service-role/StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x

Step Functions - Workflow

{
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:validateOrderFunction",
      "Next": "SaveOrderToDatabase",
      "ResultPath": "$.validationOutput"
    },
    "SaveOrderToDatabase": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:saveOrderFunction",
      "Parameters": {
        "OrderId.$": "$.validationOutput.OrderId",
        "customerEmail.$": "$.validationOutput.customerEmail",
        "productId.$": "$.validationOutput.productId",
        "quantity.$": "$.validationOutput.quantity"
      },
      "Next": "ProcessPayment"
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:processPaymentFunction",
      "Next": "UpdateInventory"
    },
    "UpdateInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:updateInventoryFunction",
      "Next": "SendNotification"
    },
    "SendNotification": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:sendNotificationFunction",
      "Next": "GenerateReceipt"
    },
    "GenerateReceipt": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:202533534284:function:generateReceiptFunction",
      "End": true
    }
  }
}

API Gateway

Step 1.2: Create API Gateway to Trigger Lambda

1. Go to the API Gateway Console.

2. Click Create API > REST API.

   • API Name: OrderProcessingAPI

   • Description: API for triggering order placement workflow.

3. Create a Resource and Method:

   • Resource: /place-order

   • Method: POST

   • Integration Type: Lambda Function

   • Lambda Function: Select OrderPlacementFunction.

4. Enable CORS:

   • Enable CORS on the /place-order method to allow cross-origin requests from the frontend.

5. Deploy the API:

   • Go to Actions > Deploy API.

   • Stage Name: prod.

6. Once deployed, you’ll get a public API URL that the frontend can use to place orders.

Create POST Method

Enable CORS

Deploy API - dev stage

Invoke URL

https://88ax43nqed.execute-api.us-east-1.amazonaws.com/dev

Go to Frontend static web app code and place the invoke url

Re deploy the frontend code to the s3 static web application bucket

npm run build

Sync build files to S3:

aws s3 sync ./build s3://your-bucket-name

Go to Static web app URL

Access the React App

S3 - static webapp

1. Frontend Deployment on S3 with CloudFront (React App)

Step 1.1: Build the React Application

1. Navigate to the frontend directory where your React project is located:

    cd /path-to-your-frontend
    Install dependencies (if you haven’t already):
   

    npm install
   

2. Build the project for production:

    npm run build
   

   This will create a build directory with static files optimized for production.

Step 1.2: Create an S3 Bucket for Static Website Hosting

1. Go to the S3 Console and click on Create Bucket.

   • Bucket Name: Choose a globally unique name (e.g., my-frontend-bucket).

   • Region: Choose a region close to your users.

   • Block all public access: Uncheck this setting to allow public access (since this is a static website).

2. Enable Static Website Hosting:

   • Go to the Properties tab.

   • Scroll down to Static website hosting.

   • Choose Enable.

   • Enter the index document as index.html and error document as index.html (for single-page apps).

3. Upload the Build Files:

   • Click on Upload.

   • Drag and drop the contents of the build folder into the S3 bucket.

4. Set Object Permissions:

   • Select the uploaded objects, go to the Permissions tab, and ensure they have public-read access for static website hosting.

5. Optional: Set up CloudFront for CDN (for faster access):

   • Go to the CloudFront Console.

   • Create a new CloudFront Distribution.

   • For Origin Domain, select your S3 bucket.

   • Use HTTPS for security.

Step 1.3: Add Permissions for S3

To make the bucket public:

1. Go to Bucket Permissions.

2. Add a Bucket Policy:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "PublicReadGetObject",
          "Effect": "Allow",
          "Principal": "*",
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::my-frontend-bucket/*"
        }
      ]
    }
   

Your React frontend is now deployed on S3.

You can access it using the S3 website URL or CloudFront distribution URL.

Challenges & Troubleshooting

Error:

POST https://88ax43nqed.execute-api.us-east-1.amazonaws.com/dev/place-order 502 (Bad Gateway)

Error

Error

App.js:8 POST https://88ax43nqed.execute-api.us-east-1.amazonaws.com/dev/place-order 500 (Internal Server Error)

App.js:16

1. {message: 'Failed to start order processing', error: "'STATE_MACHINE_ARN'"}

   1. error: "'STATE_MACHINE_ARN'"

   2. message: "Failed to start order processing"

   3. [[Prototype]]: Object

Fix: Updated the ENV variable of STATE_MACHINE_ARN

arn:aws:states:us-east-1:202533534284:stateMachine:OrderProcessingStateMachine

Re test Successful:

Error: The frontend seems to be successful but the products data does not sent to the database

Let’s investigate the error

1. Verify and ensure that the A-orderPlacement.py Lambda function is correct and it has integrated the Step Machine correctly, defined the ENV for state Machine ARN

2. Verify that the STEP Machine workflow successfully triggered

Error:

Cloudwatch

Error

2024-10-16T18:48:31.922Z

Error validating order: An error occurred (ValidationException) when calling the GetItem operation: 1 validation error detected: Value ' Inventory' at 'tableName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+

Fix:

1. Check your environment variable for the DynamoDB table name: The error shows that the table name is ' Inventory', which suggests that there is an unintended leading space in the value of the INVENTORY_TABLE environment variable.

2. Ensure that the environment variable is set correctly: Go to the AWS Lambda console and verify that the INVENTORY_TABLE environment variable is correctly set without any extra spaces.

How to Correct the Table Name:

1. Remove the Leading Space:

   • In the Lambda function configuration, under Environment Variables, find the INVENTORY_TABLE variable and remove any leading or trailing spaces.

2. Verify the Code:

   • Ensure that in the code, you're using the correct environment variable without any modifications that might introduce spaces.

Error:

Error validating order: '>=' not supported between instances of 'int' and 'str'

Item queried successful from dynamodb but still getting an error

Fix:

You need to explicitly convert the stock value retrieved from DynamoDB to an integer before comparing it with the quantity.

Here’s the updated code:

import json
import os
import boto3

# Initialize the DynamoDB client
dynamodb = boto3.client('dynamodb')

def lambda_handler(event, context):
    try:
        # Log the entire event to inspect the input
        print(f"Received event: {event}")

        # Get productId and quantity from the event
        product_id = event.get('productId')
        quantity = event.get('quantity')

        if not product_id or not quantity:
            raise Exception("Invalid input: productId and quantity are required")

        # Define the parameters for fetching the product information from DynamoDB
        table_name = os.environ['INVENTORY_TABLE'].strip()  # Strip any extra spaces
        params = {
            'TableName': table_name,
            'Key': {
                'productId': {'S': product_id}
            }
        }

        # Get the item from DynamoDB
        result = dynamodb.get_item(**params)

        # Debugging log for DynamoDB response
        print(f"DynamoDB get_item result: {result}")

        # Check if the item exists
        if 'Item' not in result:
            raise Exception(f"Product with productId {product_id} not found")

        # Check if 'stock' exists in the item and is valid
        if 'stock' not in result['Item']:
            raise Exception(f"Stock information missing for productId {product_id}")

        # Convert stock to integer for comparison
        stock = int(result['Item']['stock']['N'])

        print(f"Stock for productId {product_id}: {stock}, Requested quantity: {quantity}")

        # Ensure that quantity is an integer
        if not isinstance(quantity, int):
            quantity = int(quantity)

        # Check if stock is enough
        if stock >= quantity:
            return {
                'status': 'VALID',
                'productId': product_id,
                'quantity': quantity
            }
        else:
            raise Exception('Out of stock')

    except Exception as e:
        print(f"Error validating order: {e}")
        raise Exception('Order validation failed')

Key Changes:

1. Convert stock to an integer:

   • stock = int(result['Item']['stock']['N']) ensures that the stock value retrieved from DynamoDB is converted from a string to an integer.

2. Ensure quantity is an integer:

   • Added a check to ensure that the quantity is also an integer. If it's a string, it will be converted to an integer using quantity = int(quantity).

Redeploy the Lambda function and test

Test

Error:

Step: ValidateOrder Passed,

saveOrdertoDatabase Failed

{ "cause": "User: arn:aws:sts::202533534284:assumed-role/StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x/HyBZgMwbOpRHNcXcgWnobLmYTRXnIfkF is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:202533534284:function:saveOrderFunction because no identity-based policy allows the lambda:InvokeFunction action (Service: Lambda, Status Code: 403, Request ID: ac1acb0b-b064-4e2d-a7d8-c8a6e49709cc)", "error": "Lambda.AWSLambdaException" }

The error you're encountering is a permissions issue, where the Step Functions role does not have permission to invoke the specified Lambda function (saveOrderFunction). The role that Step Functions assumes (StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x) needs the lambda:InvokeFunction permission.

Steps to Fix:

1. Update the IAM Role for Step Functions: You need to attach a policy to the IAM role (StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x) that allows it to invoke the Lambda function (saveOrderFunction).

Option 1: Attach the Policy via the AWS Console

1. Go to the IAM Console in AWS.

2. Find the role named StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x.

3. Click Attach Policies.

4. Create a new inline policy by clicking Add permissions > Create inline policy.

5. In the JSON tab, use the following policy to allow lambda:InvokeFunction on the saveOrderFunction:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:us-east-1:202533534284:function:saveOrderFunction"
        }
    ]
}

6. Review and Save the policy.

Option 2: Attach the Policy Using the AWS CLI

You can also use the AWS CLI to attach the required permission to the IAM role.

1. Run the following command to attach the inline policy to the Step Functions role:

aws iam put-role-policy \
    --role-name StepFunctions-OrderProcessingStateMachine-role-7xpccmy1x \
    --policy-name AlowInvokeLambda \
    --policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "lambda:InvokeFunction",
                "Resource": "arn:aws:lambda:us-east-1:202533534284:function:saveOrderFunction"
            }
        ]
    }'

Explanation:

• The Action: "lambda:InvokeFunction" allows the role to invoke the Lambda function.

• The Resource specifies the ARN of the Lambda function (saveOrderFunction) that the role should be allowed to invoke.

Error: Need to pass the output of one step function to input to next step function

The error message "'OrderId' is required but not found in the event or is None" suggests that the OrderId field is either missing or None in the event passed to the saveOrderToDatabase Lambda function. This could mean that the OrderId is not being passed correctly from the previous step in the AWS Step Functions workflow.

Steps to Diagnose and Fix:

1. Verify the Event in CloudWatch Logs:

   • Ensure that the event being passed to the saveOrderToDatabase function contains the OrderId field. Add logging to capture the full event, as you are already doing with print(f"Received event: {event}").

2. Ensure OrderId is Passed Between Steps:

   • If the OrderId is being generated in a previous step, you need to make sure it is being passed correctly between steps in the Step Functions workflow.

   • In the Step Functions console, check the input and output of each state to ensure that OrderId is included in the output of the previous step and passed into this step.

Adjusting Step Functions Configuration:

If OrderId is missing because it is not passed correctly between steps, you need to make sure that the state machine's "InputPath", "ResultPath", or "OutputPath" is correctly set to pass the OrderId from one step to the next.

Example of Passing Data Between Steps:

In your Step Function definition, you can use "Parameters" to ensure the necessary fields like OrderId, customerEmail, productId, and quantity are passed to the saveOrderToDatabase step.

{
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:validateOrderFunction",
      "Next": "SaveOrderToDatabase",
      "ResultPath": "$.validationOutput"
    },
    "SaveOrderToDatabase": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:saveOrderToDatabaseFunction",
      "Parameters": {
        "OrderId.$": "$.validationOutput.OrderId",
        "customerEmail.$": "$.validationOutput.customerEmail",
        "productId.$": "$.validationOutput.productId",
        "quantity.$": "$.validationOutput.quantity"
      },
      "End": true
    }
  }
}

In this example:

• ResultPath stores the output of ValidateOrder in $.validationOutput.

• Parameters ensure that the OrderId, customerEmail, productId, and quantity are passed to the SaveOrderToDatabase function from the previous state.

Error:

{ "statusCode": 500, "body": "{\"message\": \"Payment processing failed\", \"error\": \"Missing 'amount' in the event\"}" }

Steps to Diagnose and Fix:

1. Check the Input to the Payment Lambda Function:

   • Ensure that the event passed to the Payment Processing Lambda function contains the required amount field.

   • You can log the event in the Lambda function to verify the incoming data.

2. Ensure amount is Passed Between Steps in Step Functions:

   • Make sure that the amount field is being included in the output of the previous steps and passed correctly to the Process Payment step.

Example Lambda Function with Logging:

You can log the entire event in the processPaymentFunction Lambda function to inspect the incoming event and ensure the amount is present:

import json

def lambda_handler(event, context):
    try:
        # Log the entire event to inspect the input
        print(f"Received event: {event}")

        # Extract the required fields from the event
        amount = event.get('amount')
        payment_method = event.get('paymentMethod')
        order_id = event.get('OrderId')

        # Ensure 'amount', 'paymentMethod', and 'OrderId' are present
        if not amount:
            raise Exception("Missing 'amount' in the event")
        if not payment_method:
            raise Exception("Missing 'paymentMethod' in the event")
        if not order_id:
            raise Exception("Missing 'OrderId' in the event")

        # Simulate payment processing (replace this with actual payment gateway logic)
        if payment_method == 'creditCard':
            print(f"Processing payment for order {order_id}, amount: {amount}")
            return {
                'statusCode': 200,
                'body': json.dumps({
                    'status': 'SUCCESS',
                    'orderId': order_id
                })
            }
        else:
            raise Exception('Payment method not supported')

    except Exception as e:
        print(f"Error processing payment: {e}")
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'Payment processing failed',
                'error': str(e)
            })
        }

In this example, the function logs the event received to CloudWatch. If amount is missing, it raises an exception with a clear message.

Step Functions Configuration:

Ensure that the amount field is being passed from the previous step in the Step Functions workflow to the Process Payment step.

If amount is generated or calculated in a previous step, make sure that it is passed as part of the event when transitioning to the Process Payment state.

Here’s an example of how to configure the Process Payment state in Step Functions to include amount:

{
  "ProcessPayment": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:us-east-1:202533534284:function:processPaymentFunction",
    "Parameters": {
      "amount.$": "$.orderDetails.amount",  // Adjust based on where amount is in the event
      "paymentMethod.$": "$.orderDetails.paymentMethod",
      "OrderId.$": "$.orderDetails.OrderId"
    },
    "Next": "UpdateInventory"
  }
}

Key Things to Check:

• Check Input and Output in Step Functions: In the AWS Step Functions console, review the execution history to see what input is passed to each state. Verify that the amount field is present in the input to the Process Payment state.

• Log the Input in Lambda: Add the logging in your processPaymentFunction Lambda to check what fields are passed and troubleshoot why amount is missing.

• Adjust Step Functions Parameters: If amount is generated earlier in the workflow, make sure it is passed correctly from the previous state to the Process Payment state.

Example Event:

Here’s what the event should look like when passed to the processPaymentFunction Lambda:

{
  "amount": 100,
  "paymentMethod": "creditCard",
  "OrderId": "1234"
}

Conclusion:

1. Log the Input: Verify what event is passed to the Process Payment Lambda by logging it in CloudWatch.

2. Check Step Functions: Ensure that the amount field is passed correctly from the previous step to the payment processing step.

3. Adjust Parameters: If necessary, modify the Step Functions state machine definition to ensure that amount is included in the input to the payment step.

WebApp Test

Access Web App from URL

Verify that order created successful and order receipt generated and saved to the S3 bucket

Verify from the dynamoDb ‘Order’ Table - OrderNumber:2903

Verify from “Inventory” Table that the stock count is reduced

DevOps

To deploy your Order Processing System with a DevOps approach, you need to implement a streamlined CI/CD pipeline to automate the entire deployment process for both your frontend and backend components. I'll guide you through each phase of the DevOps lifecycle to deploy this project step by step, ensuring automation, scalability, and efficiency.

Here's an outline of the phases:

Phase 1: Source Code Management (Version Control)

1.1 Setup a Git Repository

• Use Git as the version control system for managing your source code.

• If you haven’t done so, initialize a Git repository in your project folder and push the code to a remote Git repository like GitHub, GitLab, or Bitbucket.

git init
git add .
git commit -m "Initial commit for order-processing-system"
git remote add origin https://github.com/your-repo-url.git
git push -u origin main

Phase 2: Continuous Integration (CI)

2.1 Setup CI for Frontend (React App)

1. Install Dependencies and Build the Frontend:

   • In your CI pipeline, define steps to install dependencies, run tests (if any), and build the frontend code.

   • Use a service like GitHub Actions, GitLab CI, or Jenkins to automate this process.

Example for GitHub Actions:

Frontend CI Pipeline

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Install Node.js
      uses: actions/setup-node@v2
      with:
        node-version: '14'
    - run: npm install
    - run: npm run build
    - name: Upload Build Artifacts
      uses: actions/upload-artifact@v2
      with:
        name: build
        path: build/

Unit Testing

To test the OrderForm component using Jest, follow the steps below. Jest is already included in your project as a dependency, so you can directly create the test files.

Step 1: Install any necessary dependencies

Since you're using React, @testing-library/react and @testing-library/jest-dom are typically used for testing React components.

You can install these if they aren't already installed:

npm install --save-dev @testing-library/react @testing-library/jest-dom

Step 2: Create a test file for OrderForm

Inside your src folder, create a __tests__ folder (if not already created) and add a file called OrderForm.test.js:

mkdir -p src/__tests__
touch src/__tests__/OrderForm.test.js

Step 3: Write Unit Tests for OrderForm

Below is an example test for OrderForm. It mocks the fetch API, tests user interactions, and checks that the order is successfully placed.

import React from 'react';
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import '@testing-library/jest-dom/extend-expect'; // For better assertion syntax
import OrderForm from '../OrderForm'; // Adjust the path if needed

// Mocking the fetch function
global.fetch = jest.fn(() =>
  Promise.resolve({
    ok: true,
    json: () => Promise.resolve({ OrderId: '1234' }),
  })
);

describe('OrderForm Component', () => {
  beforeEach(() => {
    // Reset fetch mock before each test
    fetch.mockClear();
  });

  test('renders OrderForm and displays the title', () => {
    render(<OrderForm />);
    expect(screen.getByText('AWS Serverless Order Management System')).toBeInTheDocument();
  });

  test('submits the form and displays success message with order number', async () => {
    render(<OrderForm />);

    // Enter values into form fields
    fireEvent.change(screen.getByLabelText(/Product ID/i), { target: { value: 'P001' } });
    fireEvent.change(screen.getByLabelText(/Quantity/i), { target: { value: '2' } });
    fireEvent.change(screen.getByLabelText(/Customer Email/i), { target: { value: 'test@example.com' } });

    // Click the submit button
    fireEvent.click(screen.getByText(/Place Order/i));

    // Wait for the fetch to complete and the success message to appear
    await waitFor(() => screen.getByText(/Order placed successfully!/));

    // Assert that success message is displayed
    expect(screen.getByText(/Order placed successfully!/)).toBeInTheDocument();

    // Assert that the order number is displayed
    expect(screen.getByText(/Order Number: 1234/)).toBeInTheDocument();
  });

  test('handles failed order submission', async () => {
    // Mocking fetch to return an error response
    fetch.mockImplementationOnce(() =>
      Promise.resolve({
        ok: false,
        json: () => Promise.resolve({ error: 'Failed to place order' }),
      })
    );

    render(<OrderForm />);

    // Enter values into form fields
    fireEvent.change(screen.getByLabelText(/Product ID/i), { target: { value: 'P001' } });
    fireEvent.change(screen.getByLabelText(/Quantity/i), { target: { value: '2' } });
    fireEvent.change(screen.getByLabelText(/Customer Email/i), { target: { value: 'test@example.com' } });

    // Click the submit button
    fireEvent.click(screen.getByText(/Place Order/i));

    // Wait for the fetch to complete and error message to appear
    await waitFor(() => screen.getByText(/Failed to place order/));

    // Assert that failure message is displayed
    expect(screen.getByText(/Failed to place order/)).toBeInTheDocument();
  });
});

Step 4: Run the Tests

Now, you can run the tests using the following command:

npm test

Breakdown of the Tests:

• Test 1: Renders OrderForm Component

  • This test ensures that the form renders correctly and the title "AWS Serverless Order Management System" is displayed.

• Test 2: Successfully Places an Order

  • This test simulates filling out the form, submitting it, and checks that the order is placed successfully by asserting that the success message and order number are displayed.

• Test 3: Handles Failed Order Submission

  • This test mocks a failed response from the API and checks that an error message is displayed when the form submission fails.

Step 5: Check for Code Coverage (Optional)

You can check for code coverage by running Jest with the --coverage flag:

npm test -- --coverage

This will generate a code coverage report for your tests.

Frontend CI

To add the GitHub Actions workflow to your local project in VSCode and then upload it to GitHub, follow these steps:

Step 1: Create the GitHub Actions Workflow Directory

In your local project directory (in VSCode):

1. Navigate to the root of your project.

2. Create a .github directory inside the root directory:

   • In VSCode, right-click in the explorer view and select New Folder, then name it .github.

3. Inside the .github folder, create a workflows directory:

   • Right-click again inside .github and select New Folder, then name it workflows.

Your directory structure should look like this:

/your-project
  ├── /src
  ├── /public
  ├── /node_modules
  ├── package.json
  ├── .gitignore
  └── /.github
        └── /workflows

Step 2: Add the GitHub Actions Workflow File

1. Inside the /workflows folder, create a new YAML file for the CI pipeline. You can name it something meaningful, such as ci.yml or frontend-ci.yml.

   Example:

   • Right-click on the /workflows folder, select New File, and name it frontend-ci.yml.

2. Open the frontend-ci.yml file and add the workflow configuration for Jest tests and frontend build that we created earlier.

Frontend CI Pipeline

on:
  push:
    branches:
      - master


jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required to generate OIDC token
      contents: read   # Required to read repo contents
    steps:

       # Step 1: Checkout the repository
      - name: Checkout Code
        uses: actions/checkout@v3

      # Step 2: Configure AWS Credentials
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::202533534284:role/awsGitHubActionsRole1
          aws-region: us-east-1

       # Step 3: Set up Node.js environment
      - name: Install Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '14'

      # Step 4: Install dependencies
      - name: Install dependencies
        run: npm install
        working-directory: ./frontend

      # Step 5: Build the frontend
      - name: Build frontend
        run: npm run build
        working-directory: ./frontend

      # Step 6: Upload build artifacts
      - name: Upload Build Artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build
          path: frontend/build/

      # Step 7: Deploy to S3
      - name: Deploy to S3
        run: aws s3 sync ./frontend/build s3://ordeprocess-frontend/ --delete

Step 3: Commit the Changes and Push to GitHub

Once the GitHub Actions workflow is set up locally, you need to commit and push it to your GitHub repository.

1. Stage the changes: In VSCode terminal (or your terminal of choice), run:

    git add .github/workflows/frontend-ci.yml
   

2. Commit the changes:

    git commit -m "Add CI pipeline for Jest tests and frontend build"
   

3. Push the changes to GitHub:

    git push origin main
   

Step 4: Verify the Workflow on GitHub

After pushing the changes:

1. Go to your GitHub repository.

2. Navigate to the Actions tab in the repository.

3. You should see the Frontend CI Pipeline running automatically if a push was made to the main branch.

Summary:

• Create a .github/workflows directory in your local project.

• Add a frontend-ci.yml file inside that directory with the GitHub Actions configuration.

• Commit and push the changes to GitHub.

• Verify that the pipeline is running from the Actions tab in your GitHub repository.

To store credentials securely for a CI pipeline, you typically use the CI/CD platform’s secret management system. Here's how to securely store your AWS credentials on some common CI platforms:

Configure Secrets

In GitHub Actions, secrets can be stored in GitHub Secrets. Here's how to do it:

1. Go to your GitHub repository.

2. Navigate to Settings > Secrets and variables > Actions.

3. Click on the "New repository secret" button.

4. Add your AWS credentials as secrets:

   • Name: AWS_ACCESS_KEY_ID

   • Value: Your AWS access key.

   • Add another secret for the secret key:

     • Name: AWS_SECRET_ACCESS_KEY

     • Value: Your AWS secret access key.

Now, in your GitHub Actions workflow file (.github/workflows/<your-workflow>.yml), reference the stored secrets like this:

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Error:

Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers

Error: This request has been automatically failed because it uses a deprecated version of actions/upload-artifact: v2. Learn more: https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/

Fix:

GitHub has deprecated versions v1 and v2 of the upload-artifact action. You need to update it to use v3

Solution: Update upload-artifact to version v3

# Step 6: Upload build artifacts (Updated to v3)
      - name: Upload Build Artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build
          path: build/

Error:

Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers

Fix: Solution

To integrate GitHub Actions with AWS using OpenID Connect (OIDC), you need to manually add GitHub as an OIDC provider. This involves creating a custom OIDC identity provider for GitHub in your AWS IAM settings.

Here’s how to set it up:

1. Create GitHub as an OIDC Provider:

1. Go to the IAM console in the AWS Management Console.

2. In the left sidebar, click “Identity providers.”

3. Click on “Add provider.”

4. In the Provider type, select “OpenID Connect (OIDC)”.

5. In the Provider URL, enter:

    https://token.actions.githubusercontent.com
   

6. For Audience, enter:

    sts.amazonaws.com
   

   Click “Add provider” to create the provider.

To integrate GitHub Actions with AWS using OpenID Connect (OIDC), you need to manually add GitHub as an OIDC provider. This involves creating a custom OIDC identity provider for GitHub in your AWS IAM settings.

Here’s how to set it up:

1. Create GitHub as an OIDC Provider:

1. Go to the IAM console in the AWS Management Console.

2. In the left sidebar, click “Identity providers.”

3. Click on “Add provider.”

4. In the Provider type, select “OpenID Connect (OIDC)”.

5. In the Provider URL, enter:

    https://token.actions.githubusercontent.com
   

6. For Audience, enter:

    sts.amazonaws.com
   

7. Click “Add provider” to create the provider.

2. Create or Update an IAM Role:

1. After adding GitHub as an OIDC provider, navigate to “Roles” in the IAM console.

2. Click “Create role”.

3. Select “Web identity” as the trusted entity type.

4. In the Identity provider dropdown, select the GitHub OIDC provider you created (https://token.actions.githubusercontent.com).

5. For Audience, select sts.amazonaws.com.

6. Under "Conditions," add the following condition:

    {
      "StringEquals": {
        "token.actions.githubusercontent.com:sub": "repo:<GitHub-Org-or-User>/<Repo-Name>:ref:refs/heads/<Branch-Name>"
      }
    }
   

   • Replace <GitHub-Org-or-User>, <Repo-Name>, and <Branch-Name> with your GitHub organization/user, repository name, and branch name.

7. Assign permissions like AmazonS3FullAccess or other required policies to allow access to the necessary AWS resources.

8. Complete the role creation.

3. Update Your GitHub Actions Workflow:

Once the OIDC provider and role are set up, update your GitHub Actions workflow to assume the IAM role:

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for OIDC
      contents: read

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::<ACCOUNT_ID>:role/<Role-Name>
          aws-region: us-east-1

      - name: Deploy to S3
        run: aws s3 sync ./frontend/build s3://<bucket-name>/ --delete

Key Points:

• OIDC provider URL should be https://token.actions.githubusercontent.com.

• Audience should always be set to sts.amazonaws.com.

• The IAM role should allow the sts:AssumeRoleWithWebIdentity action.

• Ensure that the trust relationship policy specifies the GitHub repository and branch to limit access.

2. Create or Update an IAM Role:

1. After adding GitHub as an OIDC provider, navigate to “Roles” in the IAM console.

2. Click “Create role”.

3. Select “Web identity” as the trusted entity type.

4. In the Identity provider dropdown, select the GitHub OIDC provider you created (https://token.actions.githubusercontent.com).

5. For Audience, select sts.amazonaws.com.

6. Under "Conditions," add the following condition:

    {
      "StringEquals": {
        "token.actions.githubusercontent.com:sub": "repo:<GitHub-Org-or-User>/<Repo-Name>:ref:refs/heads/<Branch-Name>"
      }
    }
   

   • Replace <GitHub-Org-or-User>, <Repo-Name>, and <Branch-Name> with your GitHub organization/user, repository name, and branch name.

7. Assign permissions like AmazonS3FullAccess or other required policies to allow access to the necessary AWS resources.

8. Complete the role creation.

3. Update Your GitHub Actions Workflow:

Once the OIDC provider and role are set up, update your GitHub Actions workflow to assume the IAM role:

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for OIDC
      contents: read

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::<ACCOUNT_ID>:role/<Role-Name>
          aws-region: us-east-1

      - name: Deploy to S3
        run: aws s3 sync ./frontend/build s3://<bucket-name>/ --delete

Key Points:

• OIDC provider URL should be https://token.actions.githubusercontent.com.

• Audience should always be set to sts.amazonaws.com.

• The IAM role should allow the sts:AssumeRoleWithWebIdentity action.

• Ensure that the trust relationship policy specifies the GitHub repository and branch to limit access.

Test the GitHub Actions:

Web app

Video Link: https://youtu.be/kVH8n5w6Gkw

https://youtu.be/LlCu7K45cBI

Real-Time Use Case Solution: Real-Time Chat Application for Customer Support

Solution Overview:

To solve this problem, the company deploys a real-time chat application that integrates directly into their e-commerce website. The application consists of:

• Next.js Frontend for the chat interface, allowing customers to engage with support agents or AI-driven chatbots.

• WebSocket Backend to handle bi-directional, real-time communication between customers and the support team.

• Dockerized Containers deployed on AWS Lightsail to ensure scalability and performance.

Architecture and Workflow

Components:

1. Frontend (Next.js): A user-friendly chat widget embedded in the e-commerce site where customers can enter their inquiries. It allows real-time messaging with the support backend.

2. Backend (WebSocket Server): Handles real-time communication between the frontend chat widget and either support agents or AI bots. All communication is managed through WebSocket connections, ensuring real-time message delivery without delays.

3. Docker Containers on AWS Lightsail: The frontend and backend are deployed as separate Docker containers on AWS Lightsail, ensuring high availability, security, and cost-effectiveness.

Real-Time Workflow:

1. Customer Initiates Chat: A customer browsing the e-commerce site initiates a chat session from the embedded chat widget.

2. Connection Established: The Next.js frontend establishes a WebSocket connection to the backend running on AWS Lightsail.

3. Real-Time Messaging: The customer’s queries are sent to the backend in real-time and are either directed to a human support agent or handled by an AI-driven chatbot.

4. Instant Responses: The customer receives an instant response, reducing wait time from hours (via email) to seconds. Support agents can handle multiple conversations simultaneously, further improving efficiency.

5. Scalability: During peak times, AWS Lightsail scales the container services to handle additional chat sessions without downtime.

Key Features and Benefits

1. Instant Response Time: Real-time communication ensures that customer queries are answered immediately, leading to higher customer satisfaction.

2. Scalable Solution: The system can scale to support thousands of concurrent users, especially during peak hours or events, using the elasticity of AWS Lightsail.

3. Seamless User Experience: Customers can communicate with agents without leaving the website, providing a seamless experience that keeps them engaged and more likely to complete purchases.

4. Cost Efficiency: By deploying in lightweight Docker containers on AWS Lightsail, the company minimizes infrastructure costs while maintaining high performance.

5. Operational Efficiency: The chat system enables support agents to handle multiple customers in real time, reducing the need for hiring more agents.

Technical Stack

• Frontend: Next.js for chat UI

• Backend: WebSocket server in Node.js

• Containerization: Docker

• Cloud Platform: AWS Lightsail for cost-effective container deployment

• Scalability: Autoscaling support during high-traffic periods

• Security: SSL/TLS configuration for secure WebSocket communication

The real-time chat allows customers to receive instant answers to their queries, reducing wait times significantly and increasing customer satisfaction. The system is designed to scale dynamically during peak times, ensuring smooth operations during high-traffic periods such as sales.

 Step 1: Initialize the Project
Create a new directory for the chat application and initialize both Next.js and the WebSocket server.


mkdir realtime-chat-app
cd realtime-chat-app

### 1.1. Next.js Frontend Setup
First, let's create the Next.js frontend.

npx create-next-app@latest frontend

You can go through the prompts or use flags to set up the defaults:

npx create-next-app@latest frontend --ts --eslint --src-dir --tailwin

This command sets up a TypeScript Next.js project with ESLint, `src` directory structure, and TailwindCSS for styling (optional, but highly recommended).

### 1.2. WebSocket Backend Setup
Next, let's set up the backend in a `backend` folder, which will host the WebSocket server.


mkdir backend
cd backend
npm init -y
npm install ws express cors dotenv


The `ws` package is for WebSocket implementation, `express` is for running an HTTP server, and `dotenv` is for environment variables.

### Step 2: Directory Structure

Here is the full directory structure for the app:

realtime-chat-app/
│
├── backend/
│   ├── server.js         # WebSocket backend server
│   ├── package.json      # Node.js dependencies
│   └── .env              # Environment variables for WebSocket server
│
├── frontend/
│   ├── public/           # Static files
│   ├── src/              # Application logic
│   │   ├── components/   # React components for chat UI
│   │   ├── pages/        # Next.js pages
│   │   ├── utils/        # WebSocket connection utility
│   ├── package.json      # Next.js dependencies
│   ├── tailwind.config.js # TailwindCSS configuration (optional)
│   └── .env.local        # Frontend environment variables
└── README.md             # Project documentation
```

### Step 3: Backend WebSocket Server Setup

Inside the `backend` folder, create a file `server.js`:

```javascript
// backend/server.js
const express = require('express');
const { Server } = require('ws');
const cors = require('cors');
require('dotenv').config();

const app = express();
app.use(cors());

const PORT = process.env.PORT || 8080;

// HTTP server for serving WebSocket on the same port
const server = app.listen(PORT, () => {
  console.log(`WebSocket server running on port ${PORT}`);
});

// WebSocket server
const wss = new Server({ server });

wss.on('connection', (ws) => {
  console.log('Client connected');

  ws.on('message', (message) => {
    console.log(`Received: ${message}`);

    // Broadcasting the message to all connected clients
    wss.clients.forEach((client) => {
      if (client.readyState === ws.OPEN) {
        client.send(message);
      }
    });
  });

  ws.on('close', () => {
    console.log('Client disconnected');
  });
});
```

Create a `.env` file for the backend:

```plaintext
# backend/.env
PORT=8080
```

### Step 4: Next.js Frontend Setup

Go to the `frontend` directory and update `src/pages/index.tsx` to use WebSocket.

Here’s how the basic chat interface will look:

```tsx
// frontend/src/pages/index.tsx
import { useState, useEffect } from 'react';

const ChatPage = () => {
  const [messages, setMessages] = useState<string[]>([]);
  const [input, setInput] = useState('');
  const [ws, setWs] = useState<WebSocket | null>(null);

  useEffect(() => {
    const socket = new WebSocket(process.env.NEXT_PUBLIC_WS_URL as string);
    setWs(socket);

    socket.onmessage = (event) => {
      const newMessage = event.data;
      setMessages((prevMessages) => [...prevMessages, newMessage]);
    };

    socket.onclose = () => {
      console.log('WebSocket connection closed');
    };

    return () => {
      socket.close();
    };
  }, []);

  const sendMessage = () => {
    if (ws && input.trim()) {
      ws.send(input);
      setInput('');
    }
  };

  return (
    <div className="container mx-auto p-4">
      <h1 className="text-2xl font-bold">Chat Room</h1>
      <div className="border rounded p-2 h-64 overflow-y-scroll">
        {messages.map((message, index) => (
          <div key={index} className="py-1">{message}</div>
        ))}
      </div>
      <input
        type="text"
        value={input}
        onChange={(e) => setInput(e.target.value)}
        className="border p-2 rounded w-full my-2"
        placeholder="Type a message..."
      />
      <button
        onClick={sendMessage}
        className="bg-blue-500 text-white p-2 rounded"
      >
        Send
      </button>
    </div>
  );
};

export default ChatPage;
```

This `ChatPage` component listens to WebSocket messages and renders them in a chat box. Users can type a message and send it, which will be broadcast to all connected clients via the WebSocket server.

### Step 5: Environment Variables

In the `frontend` folder, create a `.env.local` file:

```plaintext
NEXT_PUBLIC_WS_URL=ws://localhost:8080
```

### Step 6: Tailwind CSS Setup (Optional)

If you chose to use Tailwind, add it to the frontend project:

```bash
npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init
```

Update `tailwind.config.js`:

```javascript
// frontend/tailwind.config.js
module.exports = {
  content: ['./src/**/*.{js,ts,jsx,tsx}'],
  theme: {
    extend: {},
  },
  plugins: [],
};
```

Then, add the following to your `src/styles/globals.css`:

```css
@tailwind base;
@tailwind components;
@tailwind utilities;
```

### Step 7: Run the Application

#### 7.1. Start WebSocket Backend

```bash
cd backend
npm start
```

#### 7.2. Start Next.js Frontend

```bash
cd frontend
npm run dev
```

The frontend will be available at `http://localhost:3000` and the WebSocket server at `ws://localhost:8080`.

### Step 8: Testing Routes and Final App
When you visit `http://localhost:3000`, you should see a chat interface. Messages sent will appear for all connected clients, thanks to the WebSocket setup.

### Final Routes
1. **Frontend Route**: 
   - `/` (Chat interface)
2. **WebSocket Backend Route**: 
   - `ws://localhost:8080` (WebSocket connection endpoint)

With these steps, you have a fully working chat application with a real-time WebSocket backend and Next.js frontend.

Step 7: Run the Application
7.1. Start WebSocket Backend

cd backend
npm start

7.2. Start Next.js Frontend
cd frontend
npm run dev

frontend

.env

NEXT_PUBLIC_WS_URL=ws://localhost:8080

backend

.env

# backend/.env
PORT=8080

# docker-compose.yml version: '3' services: websocket-backend: build: ./backend ports: - "8080:8080" environment: - PORT=8080 restart: always nextjs-frontend: build: ./frontend ports: - "3000:3000" environment: - NEXT_PUBLIC_WS_URL=ws://websocket-backend:8080 depends_on: - websocket-backend restart: always

docker-compose up --build

open frontend web app url

http://localhost:3000

Deploy docker containers on AWS Lightsail containers

Prerequisites:

1. AWS CLI Installed: Ensure you have the AWS CLI installed. You can verify this by running:

    aws --version
    If it’s not installed, follow the AWS CLI installation guide.
   

2. AWS CLI Configured: Make sure the AWS CLI is configured with valid credentials. If not, configure it withaws configure

   You’ll need to provide your AWS Access Key ID, Secret Access Key, Region, and Output format (e.g., json).

Steps to Create an ECR Repository:

1. Create the ECR Repository: You can create an ECR repository using the following AWS CLI command:

    ecr create-repository --repository-name <repository-name> --region <region>
    Replace <repository-name> with the name of your repository and <region> with the AWS region where you want to create it. For example:
   

    ecr create-repository --repository-name my-realtime-chatapp --region us-east-1
   

2. Example Output: Upon successful creation, you should get a response like this:

    jsonCopy code{
        "repository": {
            "repositoryArn": "arn:aws:ecr:us-east-1:123456789012:repository/my-realtime-chatapp",
            "registryId": "123456789012",
            "repositoryName": "my-realtime-chatapp",
            "repositoryUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-realtime-chatapp",
            "createdAt": 1644870241.0,
            "imageTagMutability": "MUTABLE",
            "imageScanningConfiguration": {
                "scanOnPush": false
            },
            "encryptionConfiguration": {
                "encryptionType": "AES256"
            }
        }
    }
   

   Steps to Push Docker Images to ECR:

   1. Authenticate Docker to Your ECR Registry

   Before pushing images to ECR, you need to authenticate Docker to your Amazon ECR registry.

   Run this command to authenticate (replace <your-region> with your AWS region, e.g., us-east-1):

    aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.<your-region>.amazonaws.com
   

   For example, if your AWS region is us-east-1 and your account ID is 123456789012, the command would be:

    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
   

3. Amazon ECR repositories use a specific URL format for tagging images, which includes your AWS account ID, region, and repository name.

   You will need to tag both images (realtime-chatapp-test-nextjs-frontend and realtime-chatapp-test-websocket-backend) so they point to the correct ECR repository.

   Tag the Frontend Image:

    docker tag realtime-chatapp-test-nextjs-frontend:latest <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-realtime-chatapp:frontend-latest
   

   Example:

    docker tag realtime-chatapp-test-nextjs-frontend:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-realtime-chatapp:frontend-latest
   

   Tag the Backend Imagedocker tag realtime-chatapp-test-websocket-backend:latest <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-realtime-chatapp:backend-latest

   Example:

    bdocker tag realtime-chatapp-test-websocket-backend:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-realtime-chatapp:backend-latest
   

   3. Push the Docker Images to ECR

   Now that the images are tagged, you can push them to the ECR repository.

   Push the Frontend Image:

    docker push <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-realtime-chatapp:frontend-latest
   

   Example:

    docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-realtime-chatapp:frontend-latest
   

   Push the Backend Image:

    docker push <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-realtime-chatapp:backend-latest
   

   Example:

    bashCopy codedocker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-realtime-chatapp:backend-latest
   

   Verify Your Images in ECR:

   Once the push is successful, you can verify that your images have been uploaded by going to the Amazon ECR Console and checking the my-realtime-chatapp repository.

When configuring a WebSocket connection on AWS Lightsail or any other environment, WebSockets generally use either:

• ws:// for unencrypted WebSocket connections

• wss:// for encrypted WebSocket connections (WebSockets over TLS/SSL, similar to HTTPS).

Which Protocol to Use:

1. ws:// (Unencrypted WebSocket):

   • This is used for local development or when you don’t have SSL/TLS certificates configured on your server.

   • Example: ws://localhost:8080

2. wss:// (Secure WebSocket):

   • This is the recommended protocol for production environments, including AWS Lightsail.

   • WebSocket connections secured with SSL/TLS (just like HTTPS).

   • Example: wss://your-domain.com/websocket

Deploy on AWS Lighsail

Create IAM Role:

EC2-Lightsail-ECR-Route53-Role

Create a Policy:

LightSailAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:DeleteServiceLinkedRole",
                "iam:GetServiceLinkedRoleDeletionStatus"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/lightsail.amazonaws.com/AWSServiceRoleForLightsail*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CopySnapshot",
                "ec2:DescribeSnapshots",
                "ec2:CopyImage",
                "ec2:DescribeImages"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetAccountPublicAccessBlock"
            ],
            "Resource": "*"
        }
    ]
}

Setup AWS Lightsail

1. Log In to AWS Lightsail:

   • Go to AWS Lightsail.

   • Sign in with your AWS credentials.

2. Create a Lightsail Container Service:

   • Navigate to the Containers section.

   • Click on Create a container service.

   • Choose the size of the container based on your application's resource needs (start with a smaller plan if unsure).

   • Select the AWS region closest to your users.

Deploy Frontend and Backend Containers

1. Deploy Backend Container:

   • After creating the container service, select it and click Add container.

   • Under Container Name, enter a name like backend.

   • Set the Container image to your backend image (e.g., <your-username>/realtime-chat-backend:latest).

   • Set Open ports to 8080 (the port your WebSocket server listens on).

   • Click Save.

2. Deploy Frontend Container:

   • Repeat the above steps for your frontend:

     • Add a container named frontend.

     • Set the Container image to your frontend image (e.g., <your-username>/realtime-chat-frontend:latest).

     • Set Open ports to 3000 (the port your Next.js app listens on).

   • Click Save and Deploy.

Deploy both the containers within one Lightsail Container service

Access the Webapp and verify on the browser

Conclusion

By deploying a real-time chat application using Next.js, WebSockets, and AWS Lightsail containers, the e-commerce company solved their customer support issues, scaled their infrastructure efficiently, and significantly boosted both customer satisfaction and revenue. This real-time business solution is a powerful use case for modern e-commerce platforms looking to optimize user engagement and operational efficiency.

Secure User Authentication and Role-Based Access Control for a Veterinary Clinic Management System

Business Problem:

A veterinary clinic management system, similar to the Spring PetClinic application, requires secure user authentication and role-based access control (RBAC) to ensure that different users (e.g., veterinarians, pet owners, and administrators) can access the system based on their roles. The system manages sensitive information, such as medical records, owner details, and billing data, and requires the following:

1. Authentication: Ensure that only authenticated users can access the system.

2. Authorization: Restrict access to specific data and functionalities based on the user's role (veterinarian, owner, admin).

3. Token Management: Manage secure tokens for logged-in users to ensure efficient session handling.

4. Seamless Integration: The solution must integrate seamlessly with the existing system without significant rework.

The veterinary clinic seeks a scalable and secure authentication and authorization system to prevent unauthorized access and protect sensitive data.

Challenges:

1. Implementing Role-Based Access Control (RBAC): The clinic requires the ability to control which roles can access which resources. For instance:

   • Veterinarians should have access to manage pet medical records.

   • Owners should only access their pets' details.

   • Administrators should have full access to manage clinic operations.

2. Seamless Integration with Existing Systems: The clinic management system is built using Spring Boot, and any integration must fit within the existing architecture with minimal disruption.

3. Secure Token-Based Authentication: The system needs secure token-based authentication using industry-standard protocols (e.g., OAuth2, OpenID Connect) to provide robust security.

4. Centralized User Management: The clinic desires centralized identity management to handle user credentials, roles, and permissions efficiently. It must support Single Sign-On (SSO) for both web and mobile clients in the future.

Solution: Integration of Keycloak with Spring PetClinic

Keycloak is an open-source Identity and Access Management solution that solves these authentication and authorization challenges. Keycloak provides a comprehensive framework for managing users, roles, tokens, and Single Sign-On (SSO).

We implemented Keycloak as the authentication and authorization provider for the Spring PetClinic application to meet the clinic’s security requirements.

Flow Diagram

+-------------+                     +--------------+                      +--------------+
|   Browser   | ---- Login ---->     |  Keycloak    |  ---- Redirect ----> | Spring App   |
|             | <--- Auth Token ---  |              |  <--- Token -------> |              |
+-------------+                     +--------------+                      +--------------+

Prerequisites:

• Java 11 or later

• Maven

• A running Keycloak instance (you can run it locally or in Docker)

• The Spring PetClinic application source code (available on GitHub)

Step 1: Set Up Keycloak

Install Keycloak:

Run the below shell script inside the Ubuntu machine to install keycloak

#!/bin/bash

# Objective: To install Keycloak in Ubuntu Machine
# Date: 24 SEP 2024
# Author: PRAFUL PATEL
# Web: https://www.praful.cloud
# ---------------------------------------------------------------

# Variables (modify these as per your requirements)
KEYCLOAK_VERSION="25.0.6"  # Latest stable version
KEYCLOAK_USER="keycloak"
KEYCLOAK_HOME="/opt/keycloak"
DB_USER="keycloak"
DB_PASSWORD="admin"
DB_NAME="keycloak"
ADMIN_USERNAME="admin"
ADMIN_PASSWORD="admin"

# Step 1: Update system and install required packages
echo "Updating system and installing required packages..."
sudo apt update -y
sudo apt upgrade -y
sudo apt install -y openjdk-17-jdk curl wget unzip

# Step 2: Download Keycloak
echo "Downloading Keycloak..."
wget https://github.com/keycloak/keycloak/releases/download/${KEYCLOAK_VERSION}/keycloak-${KEYCLOAK_VERSION}.zip -P /tmp

# Step 3: Install Keycloak
echo "Installing Keycloak..."
sudo unzip /tmp/keycloak-${KEYCLOAK_VERSION}.zip -d /tmp/
sudo cp -r /tmp/keycloak-${KEYCLOAK_VERSION}/* ${KEYCLOAK_HOME}/


# Step 4: Create Keycloak user and set permissions
echo "Creating Keycloak user and setting permissions..."
sudo useradd -r -d ${KEYCLOAK_HOME} -s /bin/false ${KEYCLOAK_USER}
sudo chown -R ${KEYCLOAK_USER}:${KEYCLOAK_USER} ${KEYCLOAK_HOME}
sudo chmod +x ${KEYCLOAK_HOME}/bin/kc.sh
sudo chmod -R 755 ${KEYCLOAK_HOME}

# Step 5: Set up database (Optional)
echo "Setting up PostgreSQL database..."
sudo apt install -y postgresql postgresql-contrib

sudo -u postgres psql -c "CREATE DATABASE keycloak;"
sudo -u postgres psql -c "CREATE USER ${DB_USER} WITH PASSWORD '${DB_PASSWORD}';"
sudo -u postgres psql -c "CREATE DATABASE ${DB_NAME};"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE ${DB_NAME} TO ${DB_USER};"
sudo -u postgres psql -c  "ALTER DATABASE keycloak OWNER TO keycloak;"


# Step 6: Configure Keycloak to use PostgreSQL
echo "Configuring Keycloak to use PostgreSQL..."
sudo tee ${KEYCLOAK_HOME}/conf/keycloak.conf <<EOF
db=postgres
db-url=jdbc:postgresql://localhost:5432/${DB_NAME}
db-username=${DB_USER}
db-password=${DB_PASSWORD}
EOF

# Step 7: Configure Keycloak as a service
echo "Configuring Keycloak as a service..."
sudo tee /etc/systemd/system/keycloak.service <<EOF
[Unit]
Description=Keycloak Server
After=network.target

[Service]
User=keycloak
Group=keycloak
Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
Environment="KEYCLOAK_ADMIN=admin"
Environment="KEYCLOAK_ADMIN_PASSWORD=admin"
ExecStart=/opt/keycloak/bin/kc.sh start --http-port=8081 
WorkingDirectory=/opt/keycloak
Restart=on-failure
LimitNOFILE=102642
TimeoutStopSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

EOF

# Step 8: Reload systemd, start, and enable Keycloak service
echo "Enabling and starting Keycloak service..."
sudo systemctl daemon-reload
sudo systemctl enable keycloak
sudo systemctl start keycloak

 # Start the keucloak in dev environment
sudo /opt/keycloak/bin/kc.sh start-dev --http-port=8081 --verbose


# Step 9: Display Keycloak Admin Console Information
echo "Installation complete!"
echo "Access Keycloak Admin Console at: http://localhost:8081"
echo "Admin Username: ${ADMIN_USERNAME}"
echo "Admin Password: ${ADMIN_PASSWORD}"

1.1 Install Keycloak (if not already installed)

You can run Keycloak locally or using Docker:

bashCopy codedocker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:latest start-dev

This will start Keycloak in development mode. Access Keycloak at http://localhost:8080.

1.2 Create a Realm

1. Log in to the Keycloak Admin Console (default is admin/admin).

2. Click Add Realm and create a realm called spring-petclinic.

1.3 Create a Client for Spring PetClinic

1. In your spring-petclinic realm, navigate to Clients and click Create.

2. Set the Client ID to spring-petclinic and Access Type to confidential.

3. Set the Root URL to your Spring PetClinic application’s URL, e.g., http://localhost:8080.

4. After creating the client, go to the Credentials tab and copy the Client Secret. You will need this for the Spring Boot application configuration.

1.4 Create Roles

1. Navigate to Roles and create the following roles:

   • admin

   • vet

   • owner

1.5 Create Users

1. Go to Users and create users for each role. For example:

   • admin user with the admin role.

   • vet user with the vet role.

   • owner user with the owner role.

2. Assign the corresponding role to each user in the Role Mappings tab.

Step 2: Configure Spring PetClinic to Use Keycloak

You need to modify the Spring PetClinic application to authenticate with Keycloak using the Spring Security Keycloak Adapter.

2.1 Add Keycloak Dependencies to pom.xml

In the pom.xml file, add the necessary Keycloak dependencies:

xmlCopy code<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-spring-boot-starter</artifactId>
    <version>21.0.1</version> <!-- Use the latest Keycloak version -->
</dependency>

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-spring-security-adapter</artifactId>
    <version>21.0.1</version>
</dependency>

2.2 Configure Keycloak in application.properties or application.yml

If you use application.properties, add the following:

propertiesCopy codekeycloak.realm=spring-petclinic
keycloak.auth-server-url=http://localhost:8080
keycloak.resource=spring-petclinic
keycloak.credentials.secret=YOUR_CLIENT_SECRET
keycloak.ssl-required=none
keycloak.public-client=false
keycloak.principal-attribute=preferred_username
keycloak.bearer-only=false

If you use application.yml, add the following:

yamlCopy codekeycloak:
  realm: spring-petclinic
  auth-server-url: http://localhost:8080
  resource: spring-petclinic
  credentials:
    secret: YOUR_CLIENT_SECRET
  ssl-required: none
  public-client: false
  principal-attribute: preferred_username
  bearer-only: false

Replace YOUR_CLIENT_SECRET with the client secret you copied from the Keycloak client settings earlier.

2.3 Configure Spring Security

Create a security configuration class that extends KeycloakWebSecurityConfigurerAdapter to secure the endpoints and map roles:

javaCopy codeimport org.keycloak.adapters.springsecurity.KeycloakConfiguration;
import org.keycloak.adapters.springsecurity.config.KeycloakWebSecurityConfigurerAdapter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.core.session.SessionRegistryImpl;
import org.springframework.security.web.authentication.session.RegisterSessionAuthenticationStrategy;
import org.springframework.security.web.authentication.session.SessionAuthenticationStrategy;

@KeycloakConfiguration
public class SecurityConfig extends KeycloakWebSecurityConfigurerAdapter {

    @Autowired
    public void configureGlobal(AuthenticationManagerBuilder auth) {
        auth.authenticationProvider(keycloakAuthenticationProvider());
    }

    @Bean
    @Override
    protected SessionAuthenticationStrategy sessionAuthenticationStrategy() {
        return new RegisterSessionAuthenticationStrategy(new SessionRegistryImpl());
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        super.configure(http);
        http
            .authorizeRequests()
            .antMatchers("/vets/**").hasRole("vet")
            .antMatchers("/owners/**").hasRole("owner")
            .antMatchers("/admin/**").hasRole("admin")
            .anyRequest().permitAll();
    }
}

Step 3: Test the Application

1. Start Keycloak if it’s not already running.

2. Run the Spring PetClinic Application:

    ./mvnw spring-boot:run
   

3. Access the PetClinic App: Open a browser and navigate to http://localhost:8080.

   • You will be redirected to the Keycloak login page when you try to access protected routes like /vets, /owners, or /admin.

   • Log in using the corresponding role-based users you created in Keycloak.

Step 4: Test Authorization

• Log in as Admin:

  • Navigate to /admin and log in using the admin user.

• Log in as Vet:

  • Navigate to /vets and log in using the vet user.

• Log in as Owner:

  • Navigate to /owners and log in using the owner user.

Conclusion

By following these steps, you have integrated Keycloak as the authentication and authorization provider for the Spring PetClinic application. The application now authenticates users via Keycloak and restricts access based on user roles.

Create Realm

Create Client

Create Credentials

Copy Client Secret

J4Hl5BSJYfbOAqTYmaFHL6T0MEX

Create Role

Create Roles

1. Navigate to Roles and create the following roles:

   • admin

   • vet

   • owner

Create Users

1. Go to Users and create users for each role. For example:

   • admin user with the admin role.

   • vet user with the vet role.

   • owner user with the owner role.

Role mapping

Login

Error

To Fix this error will have to integrate token controller' in spring petclinic app

To handle the Keycloak token in your Spring PetClinic application, you need to add the token management code in the appropriate parts of the Spring Boot application based on how you want to retrieve and use the token.

Here’s where and how you should add the necessary code in the Spring PetClinic application:

Configure Security in Spring Boot

1. Create a New Package:

   • Inside src/main/java/org/springframework/samples/petclinic, create a new package called security.

2. Add the SecurityConfig.java Class:

   • Inside the security package, create a file SecurityConfig.java:

Application level changes to Sprint-Petclinic Application code

Configure Security in Spring Boot

add a new package ‘security’ into java/org/springframworks/samples/petclinic
file - SecurityConfig.java

Create a New Controller to Retrieve the Token

You can create a new controller class in your Spring PetClinic application that retrieves the Keycloak token from the current session using Spring Security’s SecurityContextHolder.

Steps:

• Create a new controller file in the appropriate package (usually under org.springframework.samples.petclinic.web).

Example: TokenController.java

javaCopy codepackage org.springframework.samples.petclinic.web;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.core.oidc.user.OidcUser;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class TokenController {

    // This endpoint returns the Keycloak access token for the logged-in user
    @GetMapping("/token")
    public String getToken() {
        Authentication authentication = SecurityContextHolder.getContext().getAuthentication();

        // Check if the authenticated user is an OIDC user (Keycloak)
        if (authentication.getPrincipal() instanceof OidcUser) {
            OidcUser oidcUser = (OidcUser) authentication.getPrincipal();
            String token = oidcUser.getIdToken().getTokenValue(); // Retrieve the token
            return "Token: " + token;
        }

        return "No token available";
    }
}

• Where to put this file:

  • This file should be placed under your web package, for example, in src/main/java/org/springframework/samples/petclinic/web/TokenController.java.

Update application properties for keycloak and OAuth2

# database init, supports mysql too
database=h2
spring.sql.init.schema-locations=classpath*:db/${database}/schema.sql
spring.sql.init.data-locations=classpath*:db/${database}/data.sql

# Web
spring.thymeleaf.mode=HTML

# JPA
spring.jpa.hibernate.ddl-auto=none
spring.jpa.open-in-view=false

# Internationalization
spring.messages.basename=messages/messages

# Actuator
management.endpoints.web.exposure.include=*

# Logging
logging.level.org.springframework=INFO
# logging.level.org.springframework.web=DEBUG
# logging.level.org.springframework.context.annotation=TRACE

# Maximum time static resources should be cached
spring.web.resources.cache.cachecontrol.max-age=12h


# Keycloak config
keycloak.realm=spring-petclinic
keycloak.auth-server-url=http://localhost:8081
keycloak.resource=spring-petclinic
keycloak.credentials.secret=6HGIBPgV4yCtIvNDJFaWx110ddNNZwEg
keycloak.ssl-required=none
keycloak.public-client=false
keycloak.principal-attribute=preferred_username
keycloak.bearer-only=false
logging.level.org.springframework.security=DEBUG
logging.level.org.keycloak=DEBUG

# OAuth2 settings for Spring Security (OpenID Connect)
spring.security.oauth2.client.provider.keycloak.issuer-uri=http://localhost:8081/realms/spring-petclinic
spring.security.oauth2.client.registration.keycloak.client-id=spring-petclinic
spring.security.oauth2.client.registration.keycloak.client-secret=6HGIBPgV4yCtIvNDJFaWx110ddNNZwEg
spring.security.oauth2.client.registration.keycloak.scope=openid,profile,email
spring.security.oauth2.client.registration.keycloak.authorization-grant-type=authorization_code

Add Keycloak Dependencies to pom.xml:

1. • Open the pom.xml file and add the following dependencies:

        xmlCopy code<dependencies>
          <!-- Keycloak Integration -->
          <dependency>
            <groupId>org.keycloak</groupId>
            <artifactId>keycloak-spring-boot-starter</artifactId>
            <version>21.1.1</version>
          </dependency>
     
          <!-- Spring Security -->
          <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
          </dependency>
     
          <dependency>
            <groupId>org.springframework.security</groupId>
            <artifactId>spring-security-config</artifactId>
          </dependency>
     
          <dependency>
            <groupId>org.springframework.security</groupId>
            <artifactId>spring-security-web</artifactId>
          </dependency>
        </dependencies>
     

Add Keycloak Configuration to application.yml

Create or update the application.yml file to include Keycloak integration settings:

yamlCopy codekeycloak:
  realm: spring-petclinic
  auth-server-url: http://localhost:8080/auth
  ssl-required: external
  resource: spring-petclinic
  credentials:
    secret: YOUR_CLIENT_SECRET_HERE
  principal-attribute: preferred_username
  use-resource-role-mappings: true

spring:
  security:
    oauth2:
      client:
        registration:
          keycloak:
            client-id: spring-petclinic
            client-secret: YOUR_CLIENT_SECRET_HERE
            authorization-grant-type: authorization_code
            scope: openid, profile, email
        provider:
          keycloak:
            issuer-uri: http://localhost:8080/auth/realms/spring-petclinic

Run the Application

1. Run the Spring PetClinic application:

    ./mvnw spring-boot:run
   

2. Open the application in a browser at http://localhost:8080.

3. Attempt to access a protected route, such as /vets or /owners. You will be redirected to the Keycloak login page.

4. Log in with the corresponding Keycloak user, and you will be able to access the route based on the role assigned.

Test Token Retrieval

Access the /token endpoint to retrieve the Keycloak token:

curl http://localhost:8080/token

If authenticated, the token should be returned in the response

Test the Spring Petclinic Application with Keycloak

Login Successful with Keycloak OAuth2

Verify token

Conclusion

Integrating Keycloak with the Spring PetClinic application successfully addressed the need for secure, role-based access control and centralized identity management. By implementing token-based authentication with role mapping, we ensured that sensitive veterinary records are protected, while allowing different user groups (vets, owners, admins) to access only the data they are authorized to manage.

This solution provides the clinic with a scalable, secure, and centralized approach to identity and access management, supporting its long-term growth and security needs.

For further details on integrating Keycloak with Spring PetClinic, visit praful.cloud.

This document provides a comprehensive guide for deploying a Docker application to AWS ECS (Elastic Container Service) and ECR (Elastic Container Registry) using Jenkins. The deployment process involves building a Docker image, pushing it to ECR, updating the ECS task definition, and deploying the updated task definition to an ECS service.

Project Description: Deploying a Simple HTML Web Application on AWS ECS Using CI/CD and Load Testing

Introduction

In this project, we explore the end-to-end process of deploying a simple HTML web application using Amazon Web Services (AWS) Elastic Container Service (ECS). The deployment is facilitated by a Continuous Integration and Continuous Deployment (CI/CD) pipeline, and the application’s performance is validated through load testing. This detailed guide covers everything from setting up the necessary infrastructure to monitoring application performance, offering insights into best practices for modern cloud-native application deployment and management.

Objectives

1. Set Up Jenkins and CI/CD Pipeline:

   • Configure Jenkins on AWS EC2.

   • Set up Jenkins plugins for Docker, Amazon ECR, and pipeline management.

   • Automate the build, test, and deployment process using Jenkins.

2. Dockerize the Application:

   • Create a Dockerfile for the HTML web application.

   • Build and push Docker images to Amazon ECR.

3. Deploy on AWS ECS:

   • Configure the necessary AWS infrastructure (VPC, subnets, security groups).

   • Deploy the Dockerized application to ECS with appropriate task definitions and service configurations.

4. Load Testing and Monitoring:

   • Perform load testing using tools like Apache Benchmark (ab) and wrk.

   • Monitor application performance using Amazon CloudWatch.

   • Set up CloudWatch alarms to alert on performance issues.

Key Steps

1. Jenkins Setup and CI/CD Pipeline:

   • Installation and configuration of Jenkins.

   • Setting up Jenkins plugins for a seamless CI/CD process.

   • Writing and configuring pipeline scripts for automated deployment.

2. AWS Infrastructure Configuration:

   • Setting up VPC, subnets, and security groups to ensure proper network configuration for ECS tasks.

   • Creating and attaching IAM roles with necessary permissions for ECS services.

3. Dockerization and Deployment:

   • Writing a Dockerfile for the HTML web application.

   • Building Docker images and pushing them to Amazon ECR.

   • Configuring and deploying ECS tasks and services.

4. Performance Testing and Monitoring:

   • Conducting load testing to evaluate application performance under stress.

   • Using Amazon CloudWatch for real-time monitoring and setting up alerts for proactive issue resolution.

Outcomes

1. Automated Deployment:

   • Successful setup of a CI/CD pipeline that automates the build and deployment process.

   • Reduced manual intervention and errors, speeding up the deployment cycle.

2. Scalable and Monitored Application:

   • The HTML web application is containerized and deployed on a scalable ECS cluster.

   • Continuous performance monitoring ensures the application remains reliable and performant under load.

3. Insights from Load Testing:

   • Valuable data on how the application handles increased traffic.

   • Identified and resolved potential performance bottlenecks.

Conclusion

This project provides a comprehensive guide for deploying a simple HTML web application on AWS ECS using a CI/CD pipeline. From initial setup to performance testing and monitoring, each step is detailed to ensure a thorough understanding of the deployment process. By following this guide, you will gain hands-on experience in cloud-native application deployment, continuous integration, continuous deployment, and performance monitoring, equipping you with the skills necessary for managing modern cloud-based applications effectively.

GitHub Project Repo: https://github.com/prafulpatel16/ecs-demo.git

Prerequisites

AWS Account: Ensure you have an active AWS account. Docker: Install Docker on your local machine. AWS CLI: Install and configure the AWS CLI. AWS IAM Role: Create an IAM role with permissions for ECS, ECR, and other related services. Jenkins: Install Jenkins on your local machine or use a Jenkins server. Git Repository: Set up a Git repository (e.g., GitHub, GitLab).

Create AWS IAM Role

Create ECS Task Role Name: ecsTaskExecutionRole

Create JSON Policy: EC2ContainerRegistryReadOnly

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:DescribeRepositories",
                "ecr:GetDownloadUrlForLayer",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeImageScanFindings",
                "ecr:ListTagsForResource",
                "ecr:DescribeRegistry",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}

Prerequisites

1. AWS Account: Ensure you have an active AWS account.

2. Docker: Install Docker on your local machine.

3. AWS CLI: Install and configure the AWS CLI.

4. Code Repository: Set up a Git repository (e.g., GitHub, GitLab).

5. CI/CD Tool: Use a CI/CD tool like GitHub Actions, GitLab CI, or AWS CodePipeline.

Tools and AWS Services Used for the Project

Tools

1. Jenkins

   • Description: An open-source automation server used to automate the building, testing, and deployment of applications.

   • Usage: Set up CI/CD pipelines to automate the build and deployment process.

2. Docker

   • Description: A platform for developing, shipping, and running applications in containers.

   • Usage: Containerize the HTML web application.

3. Apache Benchmark (ab)

   • Description: A tool for benchmarking the performance of HTTP web servers.

   • Usage: Perform load testing on the deployed application.

AWS Services

1. Amazon Elastic Container Service (ECS)

   • Description: A fully managed container orchestration service.

   • Usage: Deploy and manage the containerized HTML web application.

2. Amazon Elastic Container Registry (ECR)

   • Description: A fully managed Docker container registry that makes it easy to store, manage, and deploy Docker container images.

   • Usage: Store and manage Docker images for deployment on ECS.

3. Amazon Elastic Compute Cloud (EC2)

   • Description: A web service that provides resizable compute capacity in the cloud.

   • Usage: Host Jenkins for setting up the CI/CD pipeline.

4. Amazon Virtual Private Cloud (VPC)

   • Description: A service that lets you launch AWS resources in a logically isolated virtual network.

   • Usage: Create a secure and isolated network environment for ECS tasks and services.

5. Amazon CloudWatch

   • Description: A monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers.

   • Usage: Monitor application performance, set up alarms, and gain insights into resource utilization.

6. AWS Identity and Access Management (IAM)

   • Description: A web service that helps you securely control access to AWS services and resources.

   • Usage: Manage permissions and roles for ECS tasks and services, ensuring secure access control.

7. AWS Secrets Manager

   • Description: A service to help you protect access to your applications, services, and IT resources without the upfront cost and maintenance of hardware security modules (HSMs).

   • Usage: Manage and retrieve secrets such as database credentials securely.

8. Amazon Route 53

   • Description: A scalable and highly available Domain Name System (DNS) web service.

   • Usage: Route traffic to the application deployed on ECS.

9. Amazon Simple Notification Service (SNS)

   • Description: A fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

   • Usage: Send email notifications about the status of the ECS tasks and alarms set in CloudWatch.

Additional Features

1. Email Notifications Using SNS

   • Description: Set up Amazon SNS to send email notifications for important events or alarms related to the ECS deployment.

   • Usage: Create an SNS topic, subscribe an email endpoint to the topic, and configure CloudWatch to send notifications to the SNS topic.

Implementation Steps for SNS and Email Notifications

1. Set Up SNS Topic and Email Subscription

   • Create SNS Topic:

       aws sns create-topic --name ecs-deployment-notifications
     

   • Subscribe Email to SNS Topic:

       aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications --protocol email --notification-endpoint your-email@example.com
     

2. Configure CloudWatch Alarms to Send Notifications

   • Create CloudWatch Alarm:

       aws cloudwatch put-metric-alarm --alarm-name "HighCPUUtilization" --metric-name "CPUUtilization" --namespace "AWS/ECS" --statistic "Average" --period 300 --threshold 80 --comparison-operator "GreaterThanThreshold" --dimensions Name=ClusterName,Value=your-cluster-name --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:123456789012:ecs-deployment-notifications
     

By integrating Amazon SNS and email notifications, you ensure that you receive timely updates on the status of your ECS deployments and any critical alerts, thereby enhancing the observability and reliability of your deployment process.

Deployment Steps:

1. Prepare the HTML Web App

2. Dockerize the Application

3. Create a Docker Repository on AWS ECR

4. Push the Docker Image to ECR

5. Create ECS Cluster and Task Definition

6. Create ECS Service

7. Set Up CI/CD Pipeline

8. Monitor with CloudWatch

9. Test and Validate

Implementation in Action

1. Prepare the HTML Web App

2. Dockerize the Application

# Use an official nginx image as the base image
FROM nginx:alpine

# Copy the HTML file to the nginx directory
COPY index.html /usr/share/nginx/html

# Expose port 80
EXPOSE 80

# Start nginx when the container launches
CMD ["nginx", "-g", "daemon off;"]

Build Docker Image: Build docker image in local

docker build -t simple-html-web-app .

docker image build successful

docker images

Let's the docker image locally

Start the docker image 'simple-html-web-app'

docker run -d -p 8080:80 simple-html-web-app:latest

Verify the docker image web app into Browser

Migrate the Docker application to AWS ECS Container platform

3. Create a Docker Repository on AWS ECR

Create ECR Repository:

aws ecr create-repository --repository-name simple-html-web-app

Login to ECR:

aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com

4. Push the Docker Image to ECR

docker tag simple-html-web-app:latest <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest

docker push <your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest

5. Create ECS Cluster and Task Definition

Create ECS Cluster:

aws ecs create-cluster --cluster-name simple-html-web-app-cluster

Create Task Definition:

taskdef.json

{
  "family": "simple-html-web-app-task",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "simple-html-web-app-container",
      "image": "<your-aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/simple-html-web-app:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ],
      "memory": 512,
      "cpu": 256
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::<your-aws-account-id>:role/ecsTaskExecutionRole"
}

Register Task Definition:

aws ecs register-task-definition --cli-input-json file://task-definition.json

6. Create ECS Service

Create Service:

from terminal using aws cli

aws ecs create-service \
  --cluster simple-html-web-app-cluster \
  --service-name simple-html-web-app-service \
  --task-definition simple-html-web-app-task \
  --desired-count 1 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[<your-subnet-id>],securityGroups=[<your-security-group-id>],assignPublicIp=ENABLED}"

from AWS Console

Networking: It is going to leverage default VPC and it's default security group at the moment

Create new Application Load Balancer

New ECS Service created

Set Up Jenkins CI/CD Pipeline

Jenkins installation

Create EC2 machine and Jenkins Server and install Jenkins

https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/

Part 1: Set Up Jenkins on AWS

Step 1: Launch an EC2 Instance for Jenkins

1. Sign in to AWS Management Console.

2. Launch EC2 Instance:

   • Go to Services and select EC2.

   • Click on Launch Instance.

3. Choose AMI:

   • Select an Amazon Linux 2 AMI (or Ubuntu, if preferred).

4. Choose Instance Type:

   • Select t2.medium (or another type based on your needs).

5. Configure Instance:

   • Configure VPC and Subnet.

   • Enable Auto-assign Public IP.

6. Add Storage:

   • Configure storage as needed (8GB or more).

7. Add Tags:

   • Add tags for easy identification (e.g., Key: Name, Value: Jenkins).

8. Configure Security Group:

   • Create a new security group with the following inbound rules:

     • HTTP: Port 80, Source: 0.0.0.0/0

     • Custom TCP Rule: Port 8080, Source: 0.0.0.0/0

     • SSH: Port 22, Source: 0.0.0.0/0

9. Review and Launch:

   • Review your settings and launch the instance.

   • Select or create a key pair for SSH access.

• Connect to the EC2 Instance:

  • Use SSH to connect to your EC2 instance.

    shCopy codessh -i "your-key-pair.pem" ec2-user@your-ec2-public-dns

• Install Java:

    shCopy codesudo yum update -y
    sudo amazon-linux-extras install java-openjdk11 -y
  

• Add Jenkins Repository and Install Jenkins:

    shCopy codesudo wget -O /etc/yum.repos.d/jenkins.repo \
        https://pkg.jenkins.io/redhat-stable/jenkins.repo
    sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
    sudo yum install jenkins -y
  

• Start Jenkins:

    shCopy codesudo systemctl start jenkins
    sudo systemctl enable jenkins
  

• Open Jenkins in Browser:

  • Navigate to http://your-ec2-public-dns:8080 in your browser.

  • Retrieve the initial admin password:

• 

•       shCopy codesudo cat /var/lib/jenkins/secrets/initialAdminPassword
  

• Unlock Jenkins:

  • Paste the retrieved password to unlock Jenkins.

  • Install suggested plugins during setup.

Configure Jenkins Credentials for AWS integration

Global

Add Credentials

Create a jenkins user in AWS IAM role with access key and id and provide those credentials in jenkins

Part 2: Install Required Jenkins Plugins

1. Install Plugins:

   • Go to Manage Jenkins > Manage Plugins > Available.

   • Search and install the following plugins:

     • Docker Pipeline

     • Amazon ECR

     • Pipeline

     • Git

Part 3: Configure Jenkins Pipeline for ECS

Step 1: Set Up AWS Credentials in Jenkins

1. Create AWS IAM User for Jenkins:

   • Go to IAM > Users > Add user.

   • Username: jenkins-user, Access type: Programmatic access.

   • Attach existing policies directly: AmazonEC2ContainerRegistryFullAccess, AmazonECS_FullAccess, AmazonS3FullAccess.

   • Download the Access key ID and Secret access key.

2. Configure Credentials in Jenkins:

   • Go to Manage Jenkins > Manage Credentials.

   • Under the appropriate domain (e.g., (global)), add a new AWS Credentials:

     • Kind: AWS Credentials

     • Access Key ID and Secret Access Key from the IAM user.

Step 2: Create Jenkins Pipeline

1. Create a New Pipeline Job:

   • Go to Jenkins > New Item.

   • Enter a name (e.g., ECS-Deploy-Pipeline), select Pipeline, and click OK.

2. Configure Pipeline Script:

   • In the Pipeline section, choose Pipeline script.

   • Enter the following example script:

Create a Pipline CICD

pipeline {
    agent any

    parameters {
        string(name: 'IMAGE_TAG', defaultValue: 'latest', description: 'Docker image tag')
    }

    environment {
        AWS_ACCOUNT_ID = '913151559'
        AWS_REGION = 'us-east-1'
        ECR_REPO = 'simple-html-web-app'
        ECR_REPO_URI = '9313151559.dkr.ecr.us-east-1.amazonaws.com/simple-html-web-app'
        REGISTRY_CREDENTIAL = 'aws-credentials-id' // Ensure this matches the credentials ID in Jenkins
        IMAGE_TAG = "latest-${env.BUILD_NUMBER}" // Unique tag for each build
        ECS_CLUSTER = 'simple-html-web-app-cluster'
        ECS_SERVICE = 'ecs-demo-srv'
        TASK_DEFINITION_FAMILY = 'ecs-task-def'
    }

    stages {
        stage('Test Credentials') {
            steps {
                script {
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh 'aws sts get-caller-identity'
                    }
                }
            }
        }

        stage('Clone Git Repository') {
            steps {
                checkout([$class: 'GitSCM',
                    branches: [[name: '*/master']],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [],
                    submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/prafulpatel16/ecs-demo.git']]
                ])
            }
        }

        stage('Build and Push Docker Image') {
            steps {
                script {
                    // Build and tag Docker image with specified tag
                    sh "docker build -t ${ECR_REPO_URI}:${params.IMAGE_TAG} ."

                    // Login to AWS ECR
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh """
                        aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REPO_URI}
                        """
                    }

                    // Push Docker image to ECR
                    sh "docker push ${ECR_REPO_URI}:${params.IMAGE_TAG}"
                }
            }
        }

        stage('Update ECS Task Definition') {
            steps {
                script {
                    // Load and update task definition JSON
                    def taskDefinition = readFile 'taskdef.json'
                    taskDefinition = taskDefinition.replace("REPLACE_WITH_IMAGE_TAG", "${ECR_REPO_URI}:${params.IMAGE_TAG}")

                    // Write updated task definition to file
                    writeFile file: 'taskdef.json', text: taskDefinition

                    // Register updated task definition and capture the revision number
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        def registerOutput = sh(script: "aws ecs register-task-definition --cli-input-json file://taskdef.json", returnStdout: true).trim()
                        echo "Register Output: ${registerOutput}"
                        def json = readJSON text: registerOutput
                        def taskDefinitionArn = json.taskDefinition.taskDefinitionArn
                        echo "Task Definition ARN: ${taskDefinitionArn}"
                        def taskDefinitionRevision = taskDefinitionArn.tokenize(':').last()  // Extract revision number
                        echo "Task Definition Revision: ${taskDefinitionRevision}"

                        // Save the new task definition revision for later use
                        env.TASK_DEFINITION_REVISION = taskDefinitionRevision
                    }
                }
            }
        }

        stage('Deploy ECS Service') {
            steps {
                script {
                    // Update ECS service with the new task definition revision
                    withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: "${REGISTRY_CREDENTIAL}"]]) {
                        sh """
                        aws ecs update-service --cluster ${ECS_CLUSTER} --service ${ECS_SERVICE} --task-definition ${TASK_DEFINITION_FAMILY}:${env.TASK_DEFINITION_REVISION} --force-new-deployment
                        """
                    }
                }
            }
        }
    }
}

CICD Pipeline is ready to build and deploy

Before Deployment verify the ECR repo and ECS task definition

ECR

ECS Task Definition

Let's build the code and deploy

Build Manual

Build and deploy successful

Access Web application

Hit the ALB url: http://ecs-alb-611614342.us-east-1.elb.amazonaws.com/

Verify post deployment that the new task definition has been created

Let's Change and update the code to verify that the new changes successfully apply within the CICD pipeline

Observe that on the web application there is text "Demo ECS" needs to be removed from the code and new code needs to be deployed through CICD

Go to Local VS code and remove the text

After removal the text

Commit the changes and push to the GitHub repo

git push

Expectation: New changes should be updated successfully on the web application

So if we look at the ECS Cluster definition there is 'ecs-task-def:28' which is curently running with the old web application and once the new changes is deployed it should be new 'ecs-task-def:29' should be created with the new changes.

Now let's build the code and push the changes through Jenkins CICD

Build#18

Build Successful

new 'ecs-task-def:29' deploying

deployed

Let's verify the web application has updated the changes successfully

removed text "demo-ecs"

Monitoring CloudWatch

Certainly! Here’s a comprehensive guide to performing a CPU load test on an ECS cluster web app and monitoring CPU utilization on CloudWatch using Apache Benchmark (ab) and wrk from an Amazon Linux 2 AMI:

Step 1: Set Up Amazon Linux 2 Instance

1. Launch an Amazon Linux 2 Instance:

   • Open the EC2 Dashboard in the AWS Management Console.

   • Launch a new instance and select Amazon Linux 2 AMI.

   • Choose an instance type (e.g., t2.micro).

   • Configure instance details, add storage, and configure security groups to allow SSH access.

   • Launch the instance and connect to it via SSH.

Step 2: Install Load Testing Tools on Amazon Linux 2

Apache Benchmark (ab)

1. Install Apache Benchmark:

    sudo yum update -y
    sudo yum install httpd-tools -y
   

wrk

1. Install wrk:

    sudo yum install -y git gcc
    git clone https://github.com/wg/wrk.git
    cd wrk
    make
    sudo cp wrk /usr/local/bin
   

Step 3: Perform Load Testing

Using Apache Benchmark (ab)

1. Run a Basic Load Test:

    ab -n 1000 -c 50 http://your-alb-url/
   

   • -n 1000: Number of requests to perform.

   • -c 50: Number of multiple requests to perform at a time.

   • Replace http://your-alb-url/ with your actual ALB URL.

2. Increase the Load:

    ab -n 10000 -c 200 http://your-alb-url/
   

Using wrk

1. Run a Basic Load Test:

    wrk -t12 -c400 -d30s http://your-alb-url/
   

   • -t12: Number of threads to use.

   • -c400: Number of connections to open.

   • -d30s: Duration of the test (30 seconds).

   • Replace http://your-alb-url/ with your actual ALB URL.

Step 4: Monitor CPU Utilization on CloudWatch

Enable CloudWatch Monitoring for ECS Cluster

1. Ensure CloudWatch Monitoring is Enabled:

   • Navigate to the ECS Cluster in the AWS Management Console.

   • Go to the "Monitoring" tab and ensure CloudWatch metrics are enabled.

Create CloudWatch Alarms

1. Create an Alarm for CPU Utilization:

   • Open the CloudWatch Dashboard in the AWS Management Console.

   • Click on "Alarms" > "Create Alarm".

   • Select the ECS cluster's CPUUtilization metric.

   • Set the threshold (e.g., CPU utilization > 80% for 5 minutes).

   • Configure actions (e.g., send an SNS notification).

   • Review and create the alarm.

View Metrics

1. View CPU Utilization Metrics:

   • In the CloudWatch Dashboard, navigate to "Metrics".

   • Select "ECS" and find the CPU utilization metrics for your cluster and services.

Step 5: Verify and Analyze

1. Verify Load Test Results:

   • Check the output of ab or wrk for request statistics, including requests per second, mean response time, and more.

   • Ensure your web application is handling the load as expected.

2. Analyze CloudWatch Metrics:

   • Go to the CloudWatch Dashboard and check the CPU utilization metrics.

   • Ensure that the ECS tasks are scaling properly based on the load and the alarms are triggering as expected.

Example CloudWatch Alarm JSON (Optional)

If you prefer to create the CloudWatch alarm using AWS CLI, here is an example JSON configuration:

{
  "AlarmName": "ECS-CPU-Utilization-High",
  "AlarmDescription": "Alarm when ECS CPU utilization exceeds 80%",
  "ActionsEnabled": true,
  "OKActions": [],
  "AlarmActions": [
    "arn:aws:sns:us-east-1:123456789012:MySNSTopic"
  ],
  "MetricName": "CPUUtilization",
  "Namespace": "AWS/ECS",
  "Statistic": "Average",
  "Dimensions": [
    {
      "Name": "ClusterName",
      "Value": "your-ecs-cluster-name"
    },
    {
      "Name": "ServiceName",
      "Value": "your-ecs-service-name"
    }
  ],
  "Period": 300,
  "EvaluationPeriods": 1,
  "Threshold": 80.0,
  "ComparisonOperator": "GreaterThanThreshold"
}

You can use the AWS CLI to create the alarm with the above configuration:

aws cloudwatch put-metric-alarm --cli-input-json file://alarm.json

Replace [file://alarm.json](file://alarm.json) with the path to your JSON file.

By following these steps, you can perform load testing on your ECS cluster web application and monitor its CPU utilization using CloudWatch.

Create Alarm

Select Metric

Increase the Load:

SSH to the Load test machine

ab -n 10000 -c 200 http://your-alb-url/

Hit the ALB URL 10 to 20 times and observe the load

Observer the CPU Load

ALarm Triggered

Email Notification triggered

High CPU Utlilization

Project Conclusion

Overview

The objective of this project was to deploy a simple HTML web application on AWS ECS using a CI/CD pipeline and to ensure the application’s performance under load through load testing and monitoring. This comprehensive exercise demonstrated the full lifecycle of a cloud-native application, from development and deployment to performance monitoring and scaling.

Key Steps and Achievements

1. Setting Up Jenkins and CI/CD Pipeline:

   • Installed and configured Jenkins on an AWS EC2 instance.

   • Set up necessary Jenkins plugins for Docker, Amazon ECR, and pipeline management.

   • Created a Jenkins pipeline script to build, push Docker images to ECR, and deploy to ECS.

2. Dockerization and Deployment to AWS ECS:

   • Created a Dockerfile for the simple HTML web application.

   • Built Docker images and pushed them to Amazon ECR.

   • Deployed the application to AWS ECS with appropriate task definitions and service configurations.

3. Infrastructure Setup on AWS:

   • Configured necessary VPC, subnets, and security groups to ensure the ECS tasks had proper networking configurations.

   • Created IAM roles and policies to grant necessary permissions for ECS tasks and services.

4. Load Testing and Monitoring:

   • Performed load testing using Apache Benchmark (ab) and wrk to simulate user traffic and evaluate the application's performance.

   • Monitored ECS cluster and service performance using Amazon CloudWatch.

   • Configured CloudWatch alarms to alert when CPU utilization thresholds were exceeded.

5. Troubleshooting and Optimization:

   • Resolved common issues such as task network configuration errors and IAM role permissions.

   • Tuned ECS service and task definitions for optimal performance under load.

Project Outcomes

1. Successful Deployment:

   • The HTML web application was successfully containerized and deployed on AWS ECS.

   • The CI/CD pipeline automated the process of building, testing, and deploying the application, ensuring efficient and reliable deployments.

2. Effective Load Testing:

   • Load tests provided valuable insights into the application's performance and scalability.

   • Identified and mitigated potential bottlenecks, ensuring the application could handle increased traffic.

3. Robust Monitoring and Alerts:

   • CloudWatch metrics and alarms enabled proactive monitoring of the application's performance.

   • Ensured timely alerts and response to any performance degradation or failures.

Lessons Learned

1. Automation is Key:

   • Automating the deployment process using Jenkins and ECS significantly reduces manual errors and speeds up the deployment process.

2. Importance of Monitoring:

   • Continuous monitoring and proactive alerting are crucial for maintaining application performance and availability.

3. Thorough Testing:

   • Load testing is essential to understand how the application performs under various traffic conditions and to ensure it can scale appropriately.

Future Work

1. Advanced CI/CD Features:

   • Implementing advanced CI/CD features such as blue-green deployments or canary releases to minimize downtime and reduce risk during updates.

2. Enhanced Monitoring:

   • Integrating more sophisticated monitoring and logging tools such as AWS X-Ray for distributed tracing and deeper insights into application performance.

3. Security Enhancements:

   • Implementing more robust security measures, including using AWS Secrets Manager for managing sensitive information and enhancing IAM policies.

Conclusion

This project demonstrated the end-to-end process of deploying a web application on AWS ECS, from setting up the CI/CD pipeline to ensuring the application’s performance under load. Through this exercise, key skills in cloud infrastructure management, continuous integration, continuous deployment, and performance monitoring were reinforced, providing a solid foundation for managing and scaling cloud-native applications effectively.

Key Takeaways from the Project

1. Hands-On Experience with AWS Services:

   • ECS: Learned to deploy and manage containerized applications using Amazon Elastic Container Service.

   • ECR: Managed Docker images using Amazon Elastic Container Registry.

   • EC2: Utilized Amazon EC2 instances for running Jenkins and load testing tools.

   • CloudWatch: Monitored application performance and set up alerts using Amazon CloudWatch.

   • IAM: Configured and managed IAM roles and policies for secure access and permissions.

   • SNS: Implemented Amazon SNS for email notifications on deployment status and performance alerts.

2. CI/CD Pipeline Implementation:

   • Jenkins: Set up a Jenkins server on AWS, installed necessary plugins, and created a Jenkins pipeline for automated deployments.

   • Docker: Built, managed, and deployed Docker containers, ensuring consistent application environments.

   • Automated Builds and Deployments: Configured Jenkins to automate the build and deployment process, reducing manual intervention and ensuring faster delivery.

3. Containerization Best Practices:

   • Dockerfile Creation: Created optimized Dockerfiles for the web application.

   • Docker Compose: Used Docker Compose for local development and testing of multi-container applications.

4. Monitoring and Load Testing:

   • Apache Benchmark (ab) and wrk: Conducted load tests to measure the performance and resilience of the application under high traffic.

   • CloudWatch Metrics and Alarms: Set up CloudWatch metrics and alarms to monitor CPU utilization, memory usage, and application logs.

5. Networking and Security:

   • VPC Configuration: Created and managed VPCs, subnets, and security groups to ensure secure and isolated network environments.

   • Security Groups and IAM Roles: Configured security groups to allow necessary traffic and set up IAM roles with least privilege access.

6. Handling Real-World Scenarios:

   • Troubleshooting Deployment Issues: Resolved common deployment errors such as network configuration issues and IAM role permissions.

   • Scaling and Performance Optimization: Learned to scale ECS services and optimize performance based on load testing results.

   • Automated Notifications: Implemented SNS for real-time notifications on deployment status and performance issues.

Scenarios and Experiences

1. Scenario: Automated CI/CD Pipeline with Jenkins:

   • Experience: Set up a Jenkins server on an EC2 instance, configured it with necessary plugins, and created a pipeline that automated the build, test, and deployment of a Dockerized web application to ECS. This demonstrated an understanding of continuous integration and continuous deployment best practices.

2. Scenario: Containerization and Deployment on ECS:

   • Experience: Containerized a simple HTML web application using Docker, pushed the image to Amazon ECR, and deployed it on ECS. This involved creating and configuring task definitions, clusters, and services in ECS, showcasing knowledge of container orchestration.

3. Scenario: Load Testing and Monitoring:

   • Experience: Conducted load testing using Apache Benchmark (ab) and wrk to simulate high traffic on the web application. Monitored application performance with CloudWatch and set up alarms to trigger notifications via SNS in case of performance degradation. This highlighted skills in performance testing and monitoring.

4. Scenario: Handling Deployment Errors:

   • Experience: Faced and resolved issues related to ECS service creation, IAM role permissions, and network configurations. This involved troubleshooting errors such as "ResourceInitializationError" and ensuring proper network settings for ECS tasks to access ECR. This scenario demonstrated problem-solving skills and the ability to debug deployment issues.

5. Scenario: Secure and Scalable Network Setup:

   • Experience: Configured VPCs, subnets, and security groups to create a secure and scalable network environment for the ECS cluster. This included setting up proper routing for ENIs and ensuring security group rules allowed necessary traffic. This showcased an understanding of AWS networking and security best practices.

Interview Discussion Points

• Discuss the end-to-end deployment process using Jenkins and ECS, highlighting the automation of build and deployment stages.

• Explain the benefits of containerization and how Docker was used to ensure consistent application environments.

• Describe the monitoring setup with CloudWatch and how it helped in maintaining application performance and uptime.

• Share insights on load testing results and how they influenced performance optimization and scaling decisions.

• Highlight the troubleshooting steps taken to resolve deployment issues and the importance of proper IAM and network configurations.

• Emphasize the importance of security in cloud deployments, mentioning the setup of VPCs, subnets, and security groups.

Project Description:

Our project is dedicated to crafting a resilient deployment strategy for a Java web application using Azure DevOps. We prioritize high availability and reliability by deploying the application across two distinct Azure regions: WestUS and EastUS. Additionally, we implement a primary-secondary deployment model within the development environment, bolstering fault tolerance and fortifying disaster recovery capabilities.

For more insights, visit my project blog: https://praful-cloud.gitbook.io/azure-devops/

Key Features and Components:

1. 1.Azure DevOps Pipelines: We utilize Azure DevOps Pipelines to automate the build and deployment processes of our Java web application. Through pipelines, we define tasks for building the application, running tests, and packaging the artifacts for deployment.

2. 2.Azure App Service: The Java web application is hosted on Azure App Service, a fully managed platform for building, deploying, and scaling web apps. We configure separate instances of Azure App Service in both WestUS and EastUS regions to ensure redundancy and minimize downtime.

3. 3.Release Pipelines: We establish two distinct release pipelines for deploying the application in the WestUS and EastUS regions, respectively. Each release pipeline orchestrates the deployment process, ensuring seamless delivery of updates to the application.

4. 4.Primary-Secondary Deployment: Within the development environment, we implement a primary-secondary deployment pattern to enhance resilience. This involves deploying the application to two separate environments: Dev-Primary and Dev-Secondary. In case of failures or issues in the primary environment, traffic can be automatically rerouted to the secondary environment, minimizing disruptions to development workflows.

5. 5.Service Connections: To facilitate secure integration between Azure DevOps and Azure resources, we configure service connections that provide the necessary permissions for deploying to Azure App Service instances in multiple regions.

Web App1

Web App2

Project Goals:

• High Availability: By deploying the application in two different Azure regions, we aim to ensure high availability and minimize downtime caused by region-specific failures or disruptions.

• Fault Tolerance: The primary-secondary deployment pattern enhances fault tolerance within the development environment, enabling rapid failover and recovery in case of issues.

• Automated Deployment: Through the use of Azure DevOps pipelines and release pipelines, we achieve automated deployment of the Java web application, reducing manual intervention and improving deployment efficiency.

• Resilient Development Workflow: By implementing redundancy and failover mechanisms, we create a resilient development workflow that allows developers to continue working uninterrupted, even in the event of infrastructure failures.

Conclusion:

Our project leverages Azure DevOps and Azure services to implement a multi-region deployment strategy for a Java web application, ensuring high availability, fault tolerance, and resilience. By deploying the application in two distinct regions and establishing primary-secondary deployment patterns, we enhance the reliability of the application deployment process, providing a robust foundation for development and operations teams.

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different Regions (also known as an inter-Region VPC peering connection).

VPC Peering Lifecycle A VPC peering connection goes through various stages starting from when the request is initiated. At each stage, there may be actions that you can take, and at the end of its lifecycle, the VPC peering connection remains visible in the Amazon VPC console and API or command line output for a period of time.

VPC Peering Connection Lifecycle:

Initiating-request: A request for a VPC peering connection is initiated, moving to either pending-acceptance or failed state.

Failed: The VPC peering connection request has failed. It remains visible for 2 hours and cannot be accepted, rejected, or deleted during this period.

Pending-acceptance: The request awaits acceptance from the accepter VPC owner. It can be accepted, rejected, or deleted within 7 days. If no action is taken, it expires after 7 days.

Expired: The VPC peering connection request has expired. No action can be taken, and it remains visible for 2 days to both VPC owners.

Rejected: The accepter VPC owner rejects a pending-acceptance request. It remains visible to the requester for 2 days and to the accepter for 2 hours.

Provisioning: The request has been accepted and is in the process of becoming active.

Active: The VPC peering connection is active, allowing traffic flow. It can be deleted by either VPC owner but cannot be rejected.

Deleting: Applies to an inter-Region VPC peering connection being deleted. A deletion request is submitted, and the connection transitions to deleted.

Deleted: An active connection is deleted by either owner, or a pending-acceptance request is deleted by the requester. It remains visible for a specified duration to both parties.

🚀 Features:

Inter-VPC Connectivity: VPC Peering establishes a connection between VPCs, enabling seamless communication using private IP addresses. Cross-Account Connectivity: It supports secure connections between VPCs across different AWS accounts, fostering collaboration and resource sharing. Transitive Peering: VPC Peering can be extended beyond two VPCs, creating a network topology that allows interconnectedness across multiple VPCs.

🎯 Objective: This blog aims to provide a comprehensive understanding of VPC Peering, unraveling its features and showcasing a real-time use case to highlight its practical utility.

🚀 Use Case - Real-time Web and DB VPC in Two Different Regions: Consider a scenario where a web application hosted in one AWS region necessitates real-time access to a database residing in another region. VPC Peering plays a pivotal role in establishing a secure and efficient connection between the web and database VPCs, facilitating seamless data exchange.

🔗 Solution Diagram: VPC Peering Solution Diagram

provider "aws" {
  profile = var.profile
  region  = var.region_web
  alias   = "region-web"
}

provider "aws" {
  profile = var.profile
  region  = var.region_db
  alias   = "region-db"
}


#Create VPC in us-east-1
resource "aws_vpc" "vpc_useast" {
  provider             = aws.region-web
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
    Name = "master-vpc-jenkins"
  }

}

#Create VPC in us-west-2
resource "aws_vpc" "vpc_uswest" {
  provider             = aws.region-db
  cidr_block           = "192.168.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
    Name = "worker-vpc-jenkins"
  }

}

#Initiate Peering connection request from us-east-1
resource "aws_vpc_peering_connection" "useast1-uswest-2" {
  provider    = aws.region-web
  peer_vpc_id = aws_vpc.vpc_uswest.id
  vpc_id      = aws_vpc.vpc_useast.id
  #auto_accept = true
  peer_region = var.region_db

}

#Create IGW in us-east-1
resource "aws_internet_gateway" "igw" {
  provider = aws.region-web
  vpc_id   = aws_vpc.vpc_useast.id
}

#Create IGW in us-west-2
resource "aws_internet_gateway" "igw-oregon" {
  provider = aws.region-db
  vpc_id   = aws_vpc.vpc_uswest.id
}

#Accept VPC peering request in us-west-2 from us-east-1
resource "aws_vpc_peering_connection_accepter" "accept_peering" {
  provider                  = aws.region-db
  vpc_peering_connection_id = aws_vpc_peering_connection.useast1-uswest-2.id
  auto_accept               = true
}

#Create route table in us-east-1
resource "aws_route_table" "internet_route" {
  provider = aws.region-web
  vpc_id   = aws_vpc.vpc_useast.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
  route {
    cidr_block                = "192.168.1.0/24"
    vpc_peering_connection_id = aws_vpc_peering_connection.useast1-uswest-2.id
  }
  lifecycle {
    ignore_changes = all
  }
  tags = {
    Name = "Master-Region-RT"
  }
}

#Overwrite default route table of VPC(Master) with our route table entries
resource "aws_main_route_table_association" "set-master-default-rt-assoc" {
  provider       = aws.region-web
  vpc_id         = aws_vpc.vpc_useast.id
  route_table_id = aws_route_table.internet_route.id
}
#Get all available AZ's in VPC for master region
data "aws_availability_zones" "azs" {
  provider = aws.region-web
  state    = "available"
}

#Create subnet # 1 in us-east-1
resource "aws_subnet" "subnet_1" {
  provider          = aws.region-web
  availability_zone = element(data.aws_availability_zones.azs.names, 0)
  vpc_id            = aws_vpc.vpc_useast.id
  cidr_block        = "10.0.1.0/24"
}

#Create subnet #2  in us-east-1
resource "aws_subnet" "subnet_2" {
  provider          = aws.region-web
  vpc_id            = aws_vpc.vpc_useast.id
  availability_zone = element(data.aws_availability_zones.azs.names, 1)
  cidr_block        = "10.0.2.0/24"
}


#Create subnet in us-west-2
resource "aws_subnet" "subnet_1_oregon" {
  provider   = aws.region-db
  vpc_id     = aws_vpc.vpc_uswest.id
  cidr_block = "192.168.1.0/24"
}

#Create route table in us-west-2
resource "aws_route_table" "internet_route_oregon" {
  provider = aws.region-db
  vpc_id   = aws_vpc.vpc_uswest.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw-oregon.id
  }
  route {
    cidr_block                = "10.0.1.0/24"
    vpc_peering_connection_id = aws_vpc_peering_connection.useast1-uswest-2.id
  }
  lifecycle {
    ignore_changes = all
  }
  tags = {
    Name = "Worker-Region-RT"
  }
}

#Overwrite default route table of VPC(Worker) with our route table entries
resource "aws_main_route_table_association" "set-worker-default-rt-assoc" {
  provider       = aws.region-db
  vpc_id         = aws_vpc.vpc_uswest.id
  route_table_id = aws_route_table.internet_route_oregon.id
}


#Create SG for allowing TCP/8080 from * and TCP/22 from your IP in us-east-1
resource "aws_security_group" "jenkins-sg" {
  provider    = aws.region-web
  name        = "jenkins-sg"
  description = "Allow TCP/8080 & TCP/22"
  vpc_id      = aws_vpc.vpc_useast.id
  ingress {
    description = "Allow 22 from our public IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.external_ip]
  }
  ingress {
    description = "allow anyone on port 8080"
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description = "allow traffic from us-west-2"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["192.168.1.0/24"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

#Create SG for LB, only TCP/80,TCP/443 and access to jenkins-sg
resource "aws_security_group" "lb-sg" {
  provider    = aws.region-web
  name        = "lb-sg"
  description = "Allow 443 and traffic to Jenkins SG"
  vpc_id      = aws_vpc.vpc_useast.id
  ingress {
    description = "Allow 443 from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description = "Allow 80 from anywhere for redirection"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description     = "Allow traffic to jenkins-sg"
    from_port       = 0
    to_port         = 0
    protocol        = "tcp"
    security_groups = [aws_security_group.jenkins-sg.id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

#Create SG for allowing TCP/22 from your IP in us-west-2
resource "aws_security_group" "jenkins-sg-oregon" {
  provider = aws.region-db

  name        = "jenkins-sg-oregon"
  description = "Allow TCP/8080 & TCP/22"
  vpc_id      = aws_vpc.vpc_uswest.id
  ingress {
    description = "Allow 22 from our public IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.external_ip]
  }
  ingress {
    description = "Allow traffic from us-east-1"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

variable "external_ip" {
  type    = string
  default = "0.0.0.0/0"
}

variable "profile" {
  type    = string
  default = "default"
}

variable "region_web" {
  type    = string
  default = "us-west-1"
}

variable "region_db" {
  type    = string
  default = "ca-central-1"
}

# Variables
variable "ami_web" {
  description = "AMI ID for EC2 instances"
  default = "ami-0cbd40f694b804622"
}

variable "ami_db" {
  description = "AMI ID for EC2 instances"
  default = "ami-06873c81b882339ac"
}

variable "instance_type" {
  description = "EC2 instance type"
  default     = "t2.micro"
}
# AWS EC2 Instance Key Pair
variable "instance_keypair" {
  description = "AWS EC2 Key pair that need to be associated with EC2 Instance"
  type        = string
  default     = "vpc-key.pem"
}

output "VPC-ID-US-EAST-1" {
  value = aws_vpc.vpc_useast.id
}

output "VPC-ID-US-WEST-2" {
  value = aws_vpc.vpc_uswest.id
}

output "PEERING-CONNECTION-ID" {
  value = aws_vpc_peering_connection.useast1-uswest-2.id
}
# Output IPs of Web and DB Instances
output "web_instance_ip" {
  value = aws_instance.web_instance.private_ip
}

output "db_instance_ip" {
  value = aws_instance.db_instance.private_ip
}

VPC Peering in an Action:

terraform init

terraform plan

terraform apply -auto-approve

Let's verify from the aws console

Us-west-1

web-instance

ca-central-a db-instance

Verify that peering connection is Active in US-west-1

Verify vpc peering connectin in ca-central-1

Let's verify the connection from web-instance to db-instance on private connection

db-isntance private ip: 192.168.1.245

ping from web-instance to db instance on private connection is successful

ping from db-instance to web-instance on private ip 10.0.1.25

terraform destroy -auto-approve

Conclusion: In conclusion, VPC Peering emerges as a foundational element for crafting intricate and interconnected AWS architectures. Its ability to simplify network communication, support cross-account connectivity, and enable transitive peering positions it as a versatile solution for a myriad of scenarios. As organizations navigate the complexities of modern cloud environments, VPC Peering stands as a reliable ally in building robust and scalable architectures.

Explore the power of VPC Peering to enhance the connectivity and collaboration between your AWS resources, creating a network infrastructure that aligns with the demands of today's dynamic cloud landscape.

#AWS #CloudEngineering #CloudComputing #AmazonWebServices #AWSArchitecture #DevOps #CloudSolutions #CloudSecurity #InfrastructureAsCode #AWSCertification #Serverless #AWSCommunity #TechBlogs #CloudExperts #CloudMigration #CloudOps #AWSJobs #TechIndustry #CareerInTech #InnovationInCloud #devops #cloudengineerjobs #devopsjobs #azure #gcp #oci #cloudjobs, #kubernetes

Connect with me on these platforms and stay updated with the latest in technology and development! 🚀🔗😊

🌐 Website: praful.cloud 🚀
🔗 LinkedIn: Connect with me on LinkedIn 🤝
💻 GitHub: Explore my projects on GitHub 📂
🎥 YouTube: Check out my tech tutorials on YouTube 🎬
📝 Medium: Read my tech articles on Medium 📚
🔗 Dev: Follow me on Dev for developer-centric content 🖥️

PRAFUL PATEL

Features:

Data Replication:

• Description: AWS DMS enables real-time data replication between various supported databases. Changes made in the source database are automatically reflected in the target, ensuring data consistency.

• Use Cases: Continuous data synchronization, real-time reporting, and analytics.

Database Migration:

• Description: DMS simplifies the migration of databases to and from the AWS Cloud. It supports homogeneous and heterogeneous migrations, allowing seamless transitions between different database engines.

• Use Cases: Cloud migration, database version upgrades, and platform changes.

Change Data Capture (CDC):

• Description: DMS captures and tracks changes in the source database, enabling incremental updates in the target. This feature is crucial for minimizing downtime during migrations.

• Use Cases: Minimizing downtime during migrations, supporting ongoing application operations.

Schema Conversion:

• Description: DMS assists in converting the source database schema to match the target database, ensuring compatibility during migrations between different database engines.

• Use Cases: Migrating to a different database engine, ensuring seamless data structure transitions.

Data Filtering:

• Description: DMS allows the selective migration of data based on specific criteria, reducing the need to migrate entire databases.

• Use Cases: Migrating specific subsets of data, optimizing migration bandwidth.

Task Scheduling:

• Description: DMS supports the scheduling of migration and replication tasks, allowing users to plan and automate data transfer activities.

• Use Cases: Automating routine data transfer tasks, optimizing resource utilization.

Security and Encryption:

• Description: DMS ensures the security of data during transit by supporting encryption options for data in motion.

• Use Cases: Meeting security compliance standards, protecting sensitive data during migrations.

Pre-requisite:

Before embarking on this journey, ensure you have all the necessary prerequisites in place. This includes AWS credentials, appropriate access to your Amazon RDS instance, a configured S3 bucket, and the required permissions for AWS DMS.

🎯 Objective: The primary goal of this blog post is to provide a step-by-step guide to exporting data from an Amazon RDS instance (specifically MySQL) to Amazon S3. We aim to empower users with the knowledge to automate and simplify this data export process, enabling them to harness the full potential of AWS services for their data management needs.

🚀 Use Case: Consider a scenario where organizations frequently require the export of data from their Amazon RDS MySQL database to Amazon S3. This could be for various reasons such as running analytics, creating secure backups, or facilitating seamless collaboration across different services. Our use case centers around addressing the challenges and intricacies of this data export process, offering a practical and efficient solution.

Cloud Adoption:

• Scenario: An organization is transitioning from an on-premises database to the AWS Cloud. DMS is employed to migrate the existing database to Amazon RDS with minimal downtime.

Database Version Upgrade:

• Scenario: A company is upgrading its database engine version to leverage new features and improvements. DMS is utilized to perform the upgrade seamlessly, ensuring data integrity.

Continuous Data Synchronization:

• Scenario: In a scenario where real-time data updates are critical, such as in e-commerce applications, DMS is used to replicate changes from the transactional database to a reporting database.

Data Warehousing:

• Scenario: An organization wants to consolidate data from multiple databases into a central data warehouse on Amazon Redshift. DMS facilitates the ongoing data replication for analytics purposes.

Scenario Usage:

Scenario: Migrating from an On-Premises Oracle Database to Amazon Aurora MySQL Database

1. Setup:

   • Set up source and target endpoints in AWS DMS for the Oracle database and Aurora MySQL database.

   • Configure the necessary connection details, security settings, and migration task settings.

2. Data Replication:

   • Initiate a full load of existing data from Oracle to Aurora using DMS.

   • Activate Change Data Capture (CDC) to capture ongoing changes in the Oracle database.

3. Continuous Synchronization:

   • Monitor the ongoing replication process to ensure real-time updates from Oracle to Aurora.

   • Test the synchronization by making changes in the Oracle database and verifying their timely reflection in Aurora.

4. Schema Conversion:

   • Utilize DMS schema conversion tools to handle any necessary schema transformations during the migration.

   • Ensure compatibility between Oracle and Aurora MySQL data structures.

5. Completion:

   • Once satisfied with the synchronization and migration, complete the DMS task.

   • Redirect applications to use the Aurora MySQL database as the new primary data source.

🌐 Solution Diagram:

Tools & Technologies Covered:

• AWS Cloud ☁️: Foundation for the solution, providing limitless possibilities for building and deploying applications.

• Networking, VPC, Security Group, VPC Endpoint 🏞️: Ensures secure and efficient communication between services.

• RDS (MySQL) 🗄️: Manages and maintains the MySQL database.

• S3 📤: Scalable object storage for securely storing and retrieving data.

• AWS Secret Manager 🔐: Safely stores and manages sensitive information.

• AWS DMS (Database Migration Service) 🔄: Facilitates seamless data migration between databases.

Create VPC Endpoint:

Select VPC, Subnets and Security Group:

Click to create endpoint:

Copy the S3 VPC Endpoint ID:

Endpoint ID: vpce-08ec6969fd89be2fc

Go to S3 Service:

Select the Bucket created earlier:

Click to Permission - Edit:

![Image description](https://dev-to-uploads.s3.amazonaws

.com/uploads/articles/3x5rvrg346oijvs1a308.png)

Enter the policy to policy section:

{
    "Version": "2012-10-17",
    "Id": "Access-to-bucket-using-specific-endpoint",
    "Statement": [
        {
            "Sid": "Access-to-specific-VPCE",
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:List*", "s3:Put*", "s3:Get*"],
            "Resource": [
                "arn:aws:s3:::bucket-lab-rds-export-nbtwmnxdjwpsjtdi",
                "arn:aws:s3:::bucket-lab-rds-export-nbtwmnxdjwpsjtdi/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpce": "vpce-08ec6969fd89be2fc"
                }
            }
        },
        {
            "Sid": "Access-to-specific-iam-user",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::715900322913:user/username"
            },
            "Action": ["s3:List*", "s3:Get*"],
            "Resource": [
                "arn:aws:s3:::bucket-lab-rds-export-nbtwmnxdjwpsjtdi",
                "arn:aws:s3:::bucket-lab-rds-export-nbtwmnxdjwpsjtdi/*"
            ]
        }
    ]
}

S3 Bucket policy successfully updated.

Go to Secret Manager:

Copy ARN of the stored secret:

Go to Database Migration Service:

Ensure that Replication instance is available:

Go to Migration Data > Endpoints Create Endpoint:

Endpoint created:

Create Target Endpoint:

Endpoint created:

Test Source Endpoint Connection:

Status: Testing:

Status: Successful:

Database Migration Tasks Create Task:

Task is in progress:

Task: Started:

Status: Load Complete:

Verify that "export" folder created in AWS S3 bucket:

Verify that .csv file is loaded into the S3 folder:

)

Open .csv file and verify that sample data is loaded

Conclusion: In conclusion, the efficiency of exporting data from Amazon RDS to Amazon S3 plays a pivotal role in optimizing data workflows. Leveraging the capabilities of AWS DMS, we aim to simplify and automate this process, ensuring data integrity and accessibility. Join us on this journey as we unlock the power of AWS services in enhancing your data management strategies. 🌐📈🚀

AWS Ref: https://aws.amazon.com/dms/

🌐 Website: praful.cloud 🚀
🔗 LinkedIn: Connect with me on LinkedIn 🤝
💻 GitHub: Explore my projects on GitHub 📂
🎥 YouTube: Check out my tech tutorials on YouTube 🎬
📝 Medium: Read my tech articles on Medium 📚
🔗 Dev: Follow me on Dev for developer-centric content 🖥️

Connect with me on these platforms and stay updated with the latest in Cloud/DevOps technology 🚀🔗😊

PRAFUL PATEL

AWS #CloudEngineering #CloudComputing #AmazonWebServices #AWSArchitecture #DevOps #CloudSolutions #CloudSecurity #InfrastructureAsCode #AWSCertification #Serverless #AWSCommunity #TechBlogs #CloudExperts #CloudMigration #CloudOps #AWSJobs #TechIndustry #CareerInTech #InnovationInCloud #devops #cloudengineerjobs #devopsjobs #azure #gcp #oci #cloudjobs, #kubernetes

🎯 Objective

Our primary objective is to demystify the deployment process of AWS Transit Gateway, empowering organizations to build a flexible and scalable network infrastructure. From enhanced connectivity to simplified management, our exploration aims to equip you with the knowledge to optimize your network resources and elevate your cloud experience.

🚀 Use Case

Let's dive into a real-world scenario that demonstrates the tangible benefits of AWS Transit Gateway deployment for different departments within your organization.

Sales Department: Imagine a scenario where the Sales team operates across various regions, each with its dedicated VPC. With AWS Transit Gateway, seamless communication is established between these VPCs, ensuring that sales applications and data are easily accessible. This not only enhances collaboration but also accelerates sales processes, contributing to increased efficiency and faster decision-making.

Marketing Department: For the Marketing team, running campaigns often involves diverse applications and services distributed across different VPCs. AWS Transit Gateway streamlines the connectivity, allowing marketing resources to interact effortlessly. Whether it's accessing analytics data or coordinating marketing tools, the deployment ensures a unified and efficient network, boosting overall productivity.

HR Department: In the HR domain, privacy and secure communication are paramount. AWS Transit Gateway facilitates a secure network environment, ensuring that HR-related applications and databases are interconnected with enhanced security measures. This ensures confidential employee data is transmitted securely, aligning with compliance standards and bolstering data integrity.

This use case exemplifies how AWS Transit Gateway deployment caters to the distinct needs of various departments within your organization, fostering collaboration, security, and efficiency.

#Transit Gateway Features:

Hub and Spoke Architecture:

Description: AWS Transit Gateway follows a hub-and-spoke architecture, allowing centralized connectivity and management. Global Reach:

Description: Transit Gateway supports a global network that spans multiple AWS Regions, providing a unified and scalable solution. Inter-Region Peering:

Description: Facilitates peering between Transit Gateways in different regions, enabling seamless communication across regions. VPN and Direct Connect Attachment:

Description: Connects remote networks using VPN and AWS Direct Connect, providing secure and reliable communication. Routing Control:

Description: Offers flexible and granular routing control, allowing customization of the traffic flow within the network. Network Manager Integration:

Description: Integrates with AWS Network Manager, simplifying the management of global networks and providing a centralized view. Scale Out:

Description: Scales horizontally to accommodate a growing number of VPCs and on-premises networks, ensuring scalability. Security Integration:

Description: Seamlessly integrates with AWS security features, allowing the enforcement of security policies across the network. CloudWatch Metrics and Monitoring:

Description: Provides CloudWatch metrics for monitoring network performance and enabling proactive management. Resource Group Tagging:

Description: Supports resource group tagging, making it easier to organize and manage resources within the Transit Gateway. Multicast Support:

Description: Enables multicast traffic support, allowing the transmission of data to multiple recipients simultaneously. Centralized Network Inspection:

Description: Facilitates centralized network inspection and monitoring for enhanced visibility and control. Transit Gateway Connect:

Description: Introduces Transit Gateway Connect for simplified VPN connectivity and better traffic engineering capabilities. VPC Ingress Routing:

Description: Offers VPC Ingress Routing, allowing for more granular control over the egress path of traffic leaving a VPC.

🛠️ Solution Diagram

Tools & Technologies Covered:

1. AWS cloud

2. AWS vpc

3. AWS subnets

4. AWS internet gateway

5. AWS route tables

6. AWS security groups

7. AWS EC2 machine

8. Transit Gateway

TRANSIT GATEWAY DEPLOYMENT

1. Create VPCs

2. Create Subnets

3. Create internet gateway

4. Attach internet gateway to VPCs

5. Create route tables • Subnet association • Add route entries

6. Create security groups

7. Transit gateway • Create transit gateway • Attach VPCs to transit gateway • Add routes between transit gateway and vpcs • Launch EC2 webservers in each vpcs • Test the transit gateway connectivity

Create VPCs

VPC VPC CIDR Block Availability Zone Availability Zone CIDR Block VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24

VPC-02-SALES 192.168.0.0/16 Us-east-1a 192.168.0.0/24

VPC-03-MRKT 172.10.0.0/16 Us-east-1a 172.10.0.0/24

Create VPC-01-HR

Create VPC-02-SALES

Create Subnets

VPC VPC CIDR Block Availability Zone Availability Zone CIDR Block VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24 Public-subne01-HR Us-east-1a 20.0.1.0/24 Private-subnet02-HR Us-east-1b 20.0.2.0/24 VPC-02-SALES 192.168.0.0/16 Us-east-1a 192.168.0.0/24 Public-subnet01-SALES Us-east-1a 192.168.1.0/24 Private-subnet02-SALES Us-east-1b 192.168.2.0/24 VPC-03-MRKT 172.10.0.0/16 Us-east-1a 172.10.0.0/24 Public-subnet01-MRKT Us-east-1a 172.10.1.0/24 Private-subnet02-MRKT Us-east-1b 172.10.2.0/24

VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24 Public-subne01-HR Us-east-1a 20.0.1.0/24 Private-subnet02-HR Us-east-1b 20.0.2.0/24

VPC-02-SALES 192.168.0.0/16 Us-east-1a 192.168.0.0/24 Public-subnet01-SALES Us-east-1a 192.168.1.0/24 Private-subnet02-SALES Us-east-1b 192.168.2.0/24

VPC-03-MRKT 172.10.0.0/16 Us-east-1a 172.10.0.0/24 Public-subnet01-MRKT Us-east-1a 172.10.1.0/24 Private-subnet02-MRKT Us-east-1b 172.10.2.0/24

Create Internet gateway

Internet Gateway VPC-01-HR IGW01-HR 172.10.0.0/24 VPC-02-SALES IGW02-SALES 172.10.1.0/24 VPC-03-MRKT IGW03-MRKT 172.10.2.0/24

VPC-01-HR IGW01-HR

VPC-02-SALES IGW02-SALES

Attach to VPC

VPC-03-MRKT IGW03-MRKT 172.10.2.0/24

Attached to VPC

Create Route tables

VPC Route Tables VPC-01-HR Public-RT01-HR VPC-02-SALES Public-RT02-SALES VPC-03-MRKT Public-RT03-MRKT

Edit routes – Add IGW

Subnet Association

Associate public HR subnet

Create Public-RT02-SALES

Add IGW

Subnet Association:

Associate public subnet

Create Route Public-RT03-MRKT

Add igw

Three route tables created

Create Transit gateway

Transit gateway: prafect-tgw

Create Transit gateway attachments: Attachment01: VPC01-HR

Attachment02: VPC02-SALES

Attachment02: VPC03-MRKT

Association

Propagations

Routes

Update Route Tables of VPCs VPC VPC CIDR Block Availability Zone Availability Zone CIDR Block VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24 VPC-02-SALES 192.168.0.0/16 Us-east-1a 192.168.0.0/24 VPC-03-MRKT 172.10.0.0/16 Us-east-1a 172.10.0.0/24

Add cross routes for VPC01-HR Go to Route Table : public-RT01-HR Add TGW routes for VPC02-SALES and VPC03-MRKT

VPC-02-SALES 192.168.0.0/16

VPC-03-MRKT 172.10.0.0/16

Add Cross route for VPC02-SALES

Select VPC02-SALES Add routes for VPC01-HR and VPC03-MRKT VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24 VPC-03-MRKT 172.10.0.0/16 Us-east-1a 172.10.0.0/24

Cross routes added

VPC VPC CIDR Block Availability Zone Availability Zone CIDR Block VPC-01-HR 20.0.0.0/16 Us-east-1a 20.0.0.0/24 VPC-02-SALES 192.168.0.0/16 Us-east-1a 192.168.0.0/24

Select Route table VPC-3-MRKT

Cross route added

Create Security Groups:

VPC01-HR Security Group: Public-SG01-HR

Add rule:

Rule 1: All SSH , 22, source: 0.0.0.0/0 Rule 2: All ICMP, All, source: 0.0.0.0/0

VPC02-SALES Security Group: Public-SG02-SALES

Add rules: Rule 1: All SSH , 22, source: 0.0.0.0/0 Rule 2: All ICMP, All, source: 0.0.0.0/0

VPC02-MRKT Security Group: Public-SG03-MRKT

Launch EC2 machines in each VPC’s

Repeat the same steps for VPC02-SALES & VPC03-MRKT

Launch webserver02-SALES

Launch webserver03-MRKT

All EC2 webservers launched

Verify the Connectivity from All VPC’s Webservers with each other Check connectivity via TGW

Connect to EC2 webserver01-HR from VPC01-HR Ping private IPs of VPC02-SALES – webserver02-SALES and VPC03-MRKT-webserver03-MRKT

Open webserver01-HR and connect

Click Connect

Get the Private IPs of both the webservers

Webserver02-SALES Private ip: 192.168.1.173

Test the connection: Test 1 – From VPC01-HR to VCP02-SALES Expected: The ping should return the ICMP response

Ping 192.168.1.173

Actual: The ping has returned the ICMP response

Ping successful to private ip from webserver01-HR to webserver02-SALES which are on two different departments’ VPC’s and connected through TGW

Webserver03-MRKT Private ip: 172.10.1.155

Test the connection: Test 2 – From VPC01-HR to VCP02-MRKT

Expected: The ping should return the ICMP response

Ping 172.10.1.155

Actual: The ping has returned the ICMP response

Ping successful to private ip from webserver01-HR to webserver03-MRKT which are on two different departments’ VPC’s and connected through TGW

Connect to EC2 webserver02-SALES from VPC02-SALES

Ping private IPs of VPC01-HR – webserver01-HR and VPC03-MRKT-webserver03-MRKT

Open webserver02-SALES and connect

Click to Connect

Webserver01-HR Private ip: 20.0.1.139

Test the connection: Test 1 – From VPC02-SALES to VCP01-HR

Expected: The ping should return the ICMP response

Ping 20.0.1.139

Actual: The ping has returned the ICMP response

Ping successful to private ip from webserver02-SALES to webserver01-HR which are on two different departments’ VPC’s and connected through TGW

Webserver03-MRKT Private ip: 172.10.1.155

Test the connection: Test 2 – From VPC02-SALES to VCP02-MRKT

Expected: The ping should return the ICMP response

Ping 172.10.1.155

Actual: The ping has returned the ICMP response

Ping successful to private ip from webserver02-SALES to webserver03-MRKT which are on two different departments’ VPC’s and connected through TGW

Clean up project

In conclusion, the deployment of AWS Transit Gateway stands as a pivotal solution that seamlessly connects and manages the networking needs of various departments, including Sales, Marketing, and HR. This robust infrastructure allows for efficient communication, data transfer, and collaboration across the organization's different segments. By leveraging the power of AWS Transit Gateway, businesses can achieve enhanced scalability, simplified network management, and improved overall performance. As we navigate the evolving landscape of cloud technologies, embracing solutions like AWS Transit Gateway becomes integral to fostering a more connected, streamlined, and future-ready enterprise. Here's to unlocking the full potential of network architecture and paving the way for innovation in the digital era.

Connect with me on these platforms and stay updated with the latest in technology and development! 🚀🔗😊

🌐 Website: praful.cloud 🚀
🔗 LinkedIn: Connect with me on LinkedIn 🤝
💻 GitHub: Explore my projects on GitHub 📂
🎥 YouTube: Check out my tech tutorials on YouTube 🎬
📝 Medium: Read my tech articles on Medium 📚
🔗 Dev.to: Follow me on Dev.to for cloud/devops-centric content 🖥️

#AWSCommunityBuilders #CloudEngineering #CloudComputing #AmazonWebServices #AWSArchitecture #DevOps #CloudSolutions #CloudSecurity #InfrastructureAsCode #AWSCertification #Serverless #AWSCommunity #TechBlogs #CloudExperts #CloudMigration #CloudOps #AWSJobs #TechIndustry #CareerInTech #InnovationInCloud #devops #cloudengineerjobs #devopsjobs #azure #gcp #oci #cloudjobs #kubernetes

PRAFUL PATEL

In today's rapidly evolving digital landscape, deploying web applications efficiently and ensuring optimal performance is essential for delivering a seamless user experience. This project aims to address these critical aspects by leveraging Infrastructure as Code (IAC) through Terraform, harnessing the content delivery power of AWS CloudFront, and implementing robust monitoring using New Relic.

🎯 Objective

The objective of this post is to bring and spread a knowledge about #aws cloud services, how to ? and where to consume the #aws services to solve the real world business challenges

🚀 Use Case:

Manual Deployment
Terraform (IaC) Deployment
New Relic Integration

This project aims to address these critical aspects by leveraging Infrastructure as Code (IAC) through Terraform, harnessing the content delivery power of AWS CloudFront, and implementing robust monitoring using New Relic.

Automated deployment of web applications is at the heart of this initiative, as it not only streamlines the provisioning of cloud infrastructure but also lays the foundation for future scalability across multiple cloud platforms, free from vendor lock-in constraints. By utilizing Terraform, a cloud-agnostic open-source tool, we ensure that our infrastructure can adapt to evolving requirements seamlessly.

The project also emphasizes the acceleration of web application performance through AWS CloudFront, a content delivery network (CDN) service. By strategically caching content at edge locations around the world, CloudFront enhances the user experience, minimizing latency and ensuring swift content delivery.

To maintain a vigilant eye on the performance and health of our web application, we integrate New Relic monitoring. This comprehensive tool provides us with invaluable insights into infrastructure visibility and response monitoring. With the aid of AWS Lambda functions, we securely send logs from our AWS S3 bucket to New Relic for real-time analysis.

This project not only underscores the importance of automated deployment and performance optimization but also provides a detailed, step-by-step guide for implementing these solutions. Whether you're new to these technologies or a seasoned professional, this project equips you with the knowledge and tools to harness the full potential of Terraform, CloudFront, and New Relic for your web applications.

🛠️ Solution Diagram:

The web application should deploy with the following criteria’s & requirements:

1. Requirement 1:

   • The web application should be deployed within the AWS Cloud platform.

2. Requirement 2:

   • The web application should be accessed securely.

3. Requirement 3:

   • The viewer’s experience should be seamless without any delays while accessing the web content and pages.

4. Requirement 4:

   • The web application should be accessible from around the world with a custom domain: www.prafect.link.

5. Requirement 5:

   • The web application logs should be stored in object storage and then sent to third-party monitoring solutions.

6. Requirement 6:

   • The web application should be deployed in an automated way without any vendor locking tools.

Solution:
Let’s discuss and analyze the solution for the above requirements:

1. Solution 1: AWS S3

   • From the AWS Cloud platform, there is a service called S3, which provides a service to deploy a static web application.

2. Solution 2: AWS ACM

   • AWS ACM is the solution that provides a wildcard certificate through which the web application can be secured with SSL/TLS communication.

3. Solution 3: AWS Cloudfront

   • AWS Cloudfront is the content delivery network service through which the web app content will be cached on edge locations nearby the viewers' proximity to improve the user's experience.

4. Solution 4: AWS Route 53

   • AWS Route 53 provides a DNS service through which the web application can have its own custom domain: www.prafect.link.

5. Solution 5: AWS S3

   • AWS S3 provides a service to store the logs and push the logs to an external monitoring tool. New Relic can be used, where S3 logs will be pushed to New Relic using a Lambda function.

6. Solution 6: Terraform

   • Terraform is the solution to provision the resources and infrastructure in an Infrastructure as Code (IAC) way. It's a cloud-agnostic open-source tool that can further scale up based on the requirements for multi-cloud.

Project Cost Estimation:
(Note: This cost is not any actual cost; it's just an estimation based on high-level requirements. Prices may vary based on adding and removing services based on requirements.)
Ref: AWS Pricing Calculator

Host a Static Website on AWS: Services Costs

Tools & Technologies Covered:

• AWS Cloud

• AWS S3

• AWS Certificate Manager

• AWS Cloudfront

• AWS Route 53

• AWS Lambda

• AWS CloudWatch

• New Relic (Monitoring)

• Terraform (Infrastructure as Code)

• Visual Studio Code (IDE)

• GitHub

Ref: AWS Services Documentation:
AWS S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html
AWS Cloudfront:
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/GettingStarted.SimpleDistribution.html

Pre-requisite:

1. AWS Free Tier Account

2. AWS IAM User created with programmatic access

3. AWS Route53 hosted domain

4. Visual Studio Code configured

5. Latest Terraform version installed

6. GitHub Account

7. Gitbash installed on desktop

8. New Relic account set up

9. Web application source code ready

AGENDA:
To achieve the goal of explaining the user how to configure each and every service within the console to make them familiar with the ins and outs of all service components, and then to present the automated way in which the entire web application resources are provisioned within a few minutes using Terraform (IaC).

a. Manual way web application configuration and optimizing web speed to accelerate web performance.

b. Automated way using Terraform to deploy the complete web application.

MANUAL DEPLOYMENT

1. Set up Web Application:

• Step 1: Register a Custom Domain with Route 53

• Step 2: Create Two Buckets

• Step 3: Configure Your Root Domain Bucket for Website Hosting

• Step 4: Configure Your Subdomain Bucket for Website Redirect

• Step 5: Configure Logging for Website Traffic

• Step 6: Upload Index and Website Content

• Step 7: Upload an Error Document

• Step 8: Edit S3 Block Public Access Settings

• Step 9: Attach a Bucket Policy

• Step 10: Test Your Domain Endpoint

• Step 11: Add Alias Records for Your Domain and Subdomain

• Step 12: Test the Website

2. Perform Latency Test without Cloudfront:

• a. Test from Canada

• b. Test from London

• c. Test from Singapore

3. Configure Certificate Manager to Make the Website Secure:

• a. Request a Certificate

• b. Validate - Add CNAME Records to Route53

• c. Redirection Bucket – Change the Request Server from HTTP to HTTPS

4. Accelerate Web Performance Using Cloudfront:

• a. Create Cloudfront Distribution

• b. Validate that Cloudfront Domain is Displayed on the Web

• c. Point Cloudfront Endpoint with Route53 Custom Domain

• d. Measure the Latency Test with Cloudfront

  • i. Test from Canada

  • ii. Test from London

  • iii. Test from Singapore

Implementation in Action:

1. Set up Web application

Step 1: Register a custom domain with Route 53

Step 2: Create two buckets

Main bucket: prafect.link
Second bucket: www.prafect.link

Step 3: Upload index and website content
Step 4: Upload an error document

Go to local terminal
AWS Configure : provide credentials
Go to source code directory: webfiles
aws s3 sync . s3://prafect.link

Step 5: Configure your root domain bucket for website hosting
Enable Static web

Step 6: Configure your subdomain bucket for website redirect
Bucket redirection: www.prafect.link

Step 7: Edit S3 Block Public Access settings
Buck Policy:

Step 9: Test your domain endpoint

Verify that “index.html” is displayed the web page
Access web page using endpoint
S3 web endpoint
URL : http://prafect.link.s3-website-us-east-1.amazonaws.com
index.html

Verify that “error.html” is displayed the error page
Access “error.html”

Step 10: Add alias records for your domain and subdomain
Add Custom domain with Route53
Verify that hosted zone exist

Create A record to add custom domain to web application
Prafect.link
Point out alias to AWS S3 website Endpoint

Create second alias for www.prafect.link

Step 11: Test the website
Access the web application using custom domain
www.prafect.link

1. Perform Latency test without Cloudfront a. Test from Canada b. Test from London c. Test from Singapore

Measure the performance of the web application
Certainly, here is the performance measurement presented in a table format:

Validate response header that it is coming from Amazon S3

Test 1: Access from Canada
Home Page

About US

Portfolio

Switch the location using VPN other than Canada to test the latency

Country : London

HomePage

Contact Us

Portfolio

Country: Singapore
HomePage

About Us

Portfolio

3.Configure the Certificate Manager
How to secure Web application?
1.Request a certificate
Certificate Manager
SSL Certificate
Request certificate

Status: Pending Validation

2.Create records in Route53

3.Go to Route53 and validate that two CNAME records created

SSL certificate issued successfully

Verify that web application is access using secure https
http://www.prafect.link : not secure

1. Do one change on redirection bucket and redirect to “https” Change from “http” to “https”

5.Accelerate the web performance using Cloudfront

a. Create cloudfront distribution
b. Validate that cloudfront domain is displayed on the web
c. Point the cloudfront endpoint with the Route53 custom domain
d. Measure the Latency test with Cloudfront
i. Test from Canada
ii. Test from London
iii. Test from Singapore

Create Cloudfront distribution
Origin domain: Provide AWS S3 web endpoint:
http://prafect.link.s3-website-us-east-1.amazonaws.com

Use only North America and Europe

Cloudfront deployment in progress

Cloudfront deployed

Custom origin

Validate that cloudfront domain is correctly deployed

Access web application using cloudfront url
URL: https://d3hffwkpocdftx.cloudfront.net

Security Checkpoint:

How to Secure AWS S3 web end point and restrict to access the web application allowing access through only Cloudfront web endpoint ?

Go to S3 Bucket policy
Go to Origin
Edit Origin

Replace this Bucket policy with cloudfront policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::prafect.link/"
}
] }

{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "AllowCloudFrontServicePrincipal",
"Effect": "Allow",
"Principal": {
"Service": "cloudfront.amazonaws.com"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::prafect.link/*",
"Condition": {
"StringEquals": {
"AWS:SourceArn": "arn:aws:cloudfront::914141388779:distribution/EU0BLI2K9XWK3"
}
}
}
]
}

Restrict to access from S3 web endpoint and allow only from cloudfront distribution endpoint

Access Web url:
Expected: it should block the request using S3 endpoint : PASSED

Only allowed using cloudfront endpoint

Point out Cloudfront URL to Route53

Replace the s3 web endpoint with cloudfront endpoint
Delete the record sets

Create New record set for Prafect.link
Routing Policy: Simple Routing

Create New record set for www.Prafect.link

Record sets created successfully