# 👁️‍🗨️ Kubernetes Monitoring 🚀 📊📈 **Prometheus & Grafana** 👁️‍🗨️ 1. 📈 **Visualize Cluster Information in Dashboards**: * Use Grafana to create dashboards that display essential cluster metrics, such as CPU and memory usage, node health, and pod status. 2. 📚 **Pull Custom Application Logs via Sidecar**: * Implement a sidecar container in your pods to extract custom application logs and make them available for monitoring. 3. 💻 **Create Dashboards as Code for Easy Editing**: * Opt for Infrastructure as Code (IaC) to define and manage your Grafana dashboards. This ensures easy editing, version control, and reproducibility of your monitoring setup. Embrace Prometheus and Grafana to gain valuable insights into your Kubernetes cluster's health and performance. 🌐🔍 #KubernetesMonitoring #Prometheus #Grafana 🛠️ 🚨 **Incident Response Scenario: Leaderboard Service Outage** 🚨 **Scenario:** You are a Container Engineer responsible for maintaining an e-commerce platform's critical leaderboard service. This service displays the top-selling products on the platform. Suddenly, you receive an urgent alert notifying you that the leaderboard service is down, and it's impacting the user experience. Your task is to quickly respond to this incident, diagnose the issue, and restore the service to its normal working state. **Role**: Container Engineer 🐳 **Platform**: E-commerce 🛒 **Service**: Leaderboard 🏆 📅 **Real-time Actions**: 1. **🔥 Immediate Alert Acknowledgment**: You swiftly acknowledge the alert, notifying your team that you're diving into the incident. 2. **📊 Monitoring and Logging Tools**: Access real-time data using monitoring tools like Prometheus and Grafana to gauge cluster health and resource use. 3. **🕵️ Kubernetes Cluster Status Check**: Utilize `kubectl` to confirm the Kubernetes cluster status. Ensure it's not a global cluster issue, inspect nodes and control plane components. 4. **📜 Leaderboard Service Logs**: Check service logs for error messages with `kubectl logs` and evaluate recent events. 5. **📦 Pod Inspection**: Run `kubectl get pods` to list all pods, including the leaderboard pod. Spot the "Failed" status. 6. **🔍 Troubleshooting the Pod**: Use `kubectl describe pod ` to uncover details about the failure, including resource and mounting issues. 7. **💾 Resource Check**: Review resource requests and limits in the pod configuration to avoid resource starvation. 8. **🔄 Rolling Restarts**: If issues are found, trigger a rolling restart by updating the Deployment to create fresh pods. 9. **👨‍⚕️ Health Checks**: Ensure liveness and readiness probes in the deployment are correctly configured. 10. **🌐 Integration and Network Issues**: Investigate integration and network problems within the cluster. 11. **🔌 Database Connectivity**: Verify the leaderboard service's ability to connect to the database, essential for fetching sales data. 12. **📦 Docker Image**: Confirm availability and correctness of the Docker image in the deployment configuration. 13. **🛡️ Service Checks**: Confirm the Kubernetes service correctly routes traffic and is reachable. 14. **🔄 Backup and Rollback Plan**: Maintain a rollback plan in case of prolonged issues. Consider implementing a backup mechanism for a default leaderboard. 15. **📝 Documentation and Communication**: Document all actions and updates, keeping the team and stakeholders informed. 16. **✅ Resolution and Verification**: After addressing the root cause, verify that the leaderboard service is operational and meets performance expectations. 17. **🔍 Post-Incident Analysis**: Conduct a post-incident analysis to understand the cause, document lessons learned, and implement preventive measures. In this real-time scenario, swift response, a systematic troubleshooting approach, and effective communication are vital to minimize downtime and maintain a positive user experience on the e-commerce platform. #IncidentResponse #Kubernetes #Ecommerce #ContainerEngineer ### 👀 **Monitoring Priorities** 👀 **1\. Node Health**: * 🏥 Monitor node health to ensure each node in the cluster is running smoothly. **2\. Cluster CPU/Memory Capacity**: * 💻 Keep an eye on cluster-wide CPU and memory capacity to prevent resource bottlenecks. **3\. Pod Health Checks**: * ❤️‍🩹 Implement health checks for pods to detect issues and ensure they're in a healthy state. **4\. Networking**: * 🌐 Monitor network traffic and connectivity to guarantee seamless communication between pods and services. **5\. Application Logs**: * 📋 Collect and analyze application logs for insights into app behavior and potential issues. **Objectives:** **1\. Identify Pod Configuration Error**: * 🕵️‍♂️ Identify the error within the pod's configuration causing the app malfunction. **2\. Update Pod Configuration**: * 🔄 Revise the pod's configuration to bring the app back to its expected, functioning state. Incorporating monitoring and addressing configuration issues are key elements of maintaining a healthy and operational Kubernetes environment. 🚀 Observer the working nodes ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697600579283/d48df2dc-e0d3-469b-9a2c-937784664e68.png align="center") Observe the running workloads ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697600696008/c4a1d145-002e-43f8-824c-f89a2f65e2a4.png align="center") Get more information about the pod leaderboard ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697600748548/5759751e-9bde-4d73-a392-45f48cabe033.png align="center") ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697600780022/a36be4d4-be52-4aff-b019-8906c6f317c2.png align="center") Check the logs for 'query-app' container ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601028927/bd26672d-16ce-487f-9490-b05fbaa7570d.png align="center") Some one had typo in the commands 'ech' which caused an error ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601112434/dad60069-2f33-405c-94ac-f6a1a4d5cd50.png align="center") **2\. Update Pod Configuration**: * 🔄 Revise the pod's configuration to bring the app back to its expected, functioning state. Export the leaderboard pod configurations into a leaderboard.yaml file: kubectl get pod leaderboard -o yaml > leaderboard.yaml Open the file: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601542305/0078c17a-a0a6-4921-8c71-070fce53571e.png align="center") vim leaderboard.yaml Edit the command to be echo instead of ech. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601511150/d835559d-a47f-47b0-ab91-017a58c77c85.png align="center") Save and exit the file by pressing Escape followed by wq. Attempt to update the pod: kubectl apply -f leaderboard.yaml We can't update the command key for a running pod, so you'll see an error instead. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601659302/fcbdad01-4410-4507-9a99-f96126d14427.png align="center") Delete the pod: kubectl delete pod leaderboard Confirm it's gone: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601704716/c6afc738-9cb7-4994-bbf2-d1f09e1bf6ea.png align="center") ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601739728/58a2e610-6395-42a5-830d-38e78e567d57.png align="center") kubectl get pods Re-create the pod: kubectl apply -f leaderboard.yaml Confirm it exists: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601790538/f844327e-00eb-413d-8b38-b1194644c242.png align="center") kubectl get pods Check the logs for the updated query-app container: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601830589/982237c1-a215-44fb-89da-b67df4a2b11a.png align="center") kubectl logs leaderboard -c query-app Get the pod description again: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601892661/48887f3c-71cc-4745-bfbb-7aeb9d241f88.png align="center") kubectl describe pod leaderboard ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601945889/1dc89a41-bc63-497f-9f33-b66d497d9c44.png align="center") ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1697601959279/6ee43a44-1824-46b8-9084-eb1c4a18694d.png align="center") #KubernetesMonitoring #PodConfiguration #AppMaintenance 🛠️