ELK in Kubernetes Production
Introduction
Understanding Elasticsearch and Kubernetes
Elasticsearch is a distributed, scalable search engine designed to handle large volumes of data efficiently. It’s perfect for scenarios like log aggregation, metrics analysis, and full-text search. On the other hand, Kubernetes is a container orchestrator that simplifies the deployment and management of containerized applications, providing scalability and resilience.
Why Deploy Elasticsearch on Kubernetes?
Running Elasticsearch on Kubernetes offers several advantages. Kubernetes provides easy scalability, simplified management, and consistent deployment across different environments. By leveraging Kubernetes features like StatefulSets and PersistentVolumes, you can ensure data persistence and high availability for your Elasticsearch cluster.
Elasticsearch Cluster Architecture:
The Elasticsearch cluster comprises three types of nodes:
⦁ Master nodes: Handle cluster-wide management and configuration.
⦁ Data nodes: Responsible for storing data and executing data-related operations like searches.
⦁ Client nodes: Route cluster requests to the master node and handle data-related requests to data nodes.
The recommended practice is to deploy a total of 7 pods in the Elasticsearch cluster: 3 Master node pods, 2 Data node pods, and 2 Client node pods. The diagram below illustrates the cluster architecture with these pods.
Data node pods are deployed as a StatefulSet with a headless service toe stable network identities. Master node pods are deployed as a ReplicaSet with a headless service, facilitating auto-discovery. Client node pods are also deployed as a ReplicaSet, but with an internal service that allows access to the Data nodes for read/write requests.
Memory Requirements for Deploying Elasticsearch on Kubernetes
When deploying Elasticsearch on Kubernetes, remember to allocate at least 4GB of memory to each Kubernetes node. To run smoothly, you’ll need a minimum of 7 nodes. The default size for PersistentVolumeClaims (PVCs) in each Elasticsearch Pod is 30GB, determining your block storage needs. Since pods are managed within a StatefulSet, ensure you have 30GB of storage available for each additional pod you create. Dealing with PVCs can be complex, especially outside of cloud services, as manual deletion may be required. Issues like pod startup failures often result from insufficient storage or lingering old PVCs.
Deploying a 7-Pod Elasticsearch Cluster on Kubernetes with Helm
To configure a robust Elasticsearch cluster, we follow best practices to set up 7 Pods: 3 master Pods, 2 data Pods, and 2 client Pods. This setup ensures optimal performance and reliability.
Start by running the Helm install command without any additional parameters:
helm install --name elasticsearch elastic/elasticsearchNext, upgrade the Elasticsearch Pods to increase their number and assign specific roles. Create three YAML configuration files for this purpose:
Master Pods Configuration:
# master.yaml
---
clusterName: "elasticsearch"
nodeGroup: "master"
roles:
master: "true"
ingest: "false"
data: "false"
replicas: 3Data Pods Configuration:
# data.yaml
--
clusterName: "elasticsearch"
nodeGroup: "data"
roles:
master: "false"
ingest: "true"
data: "true"
replicas: 2Clients Pods Configuration:
# client.yaml
---
clusterName: "elasticsearch"
nodeGroup: "client"
roles:
master: "false"
ingest: "false"
data: "false"
replicas: 2
service:
type: "LoadBalancer"Execute the upgrade command three times, using each YAML config file:
Rhelm upgrade --wait --timeout=600 --install \
--values ./master.yaml elasticsearch elastic/elasticsearch
helm upgrade --wait --timeout=600 --install \
--values ./data.yaml elasticsearch elastic/elasticsearch
helm upgrade --wait --timeout=600 --install \
--values ./client.yaml elasticsearch elastic/elasticsearchThe upgrade process might take a while. Once completed, verify the status of your resources:
kubectl get allYou should see output similar to this:
# Pods
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-client-0 1/1 Running 0 10m
pod/elasticsearch-client-1 1/1 Running 0 10m
pod/elasticsearch-data-0 1/1 Running 0 11m
pod/elasticsearch-data-1 1/1 Running 0 11m
pod/elasticsearch-master-0 1/1 Running 0 8m27s
pod/elasticsearch-master-1 1/1 Running 0 8m27s
pod/elasticsearch-master-2 1/1 Running 0 8m27s
# Services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-client LoadBalancer 10.245.114.89 <YOUR_IP> 9200:32366/TCP,9300:31427/TCP 10m
service/elasticsearch-client-headless ClusterIP None <NONE> 9200/TCP,9300/TCP 10m
service/elasticsearch-data ClusterIP 10.245.116.115 <NONE> 9200/TCP,9300/TCP 11m
service/elasticsearch-data-headless ClusterIP None <NONE> 9200/TCP,9300/TCP 11m
service/elasticsearch-master ClusterIP 10.245.220.94 <NONE> 9200/TCP,9300/TCP 8m27s
service/elasticsearch-master-headless ClusterIP None <NONE> 9200/TCP,9300/TCP 8m27s
service/kubernetes ClusterIP 10.245.0.1 <NONE> 443/TCP 4h5m
# StatefulSets
NAME READY AGE
statefulset.apps/elasticsearch-client 2/2 10m
statefulset.apps/elasticsearch-data 2/2 11m
statefulset.apps/elasticsearch-master 3/3 8m28s
To ensure everything is working, use curl to check the Elasticsearch endpoint:
curl http://<YOUR_IP>/_cluster/state?pretty Your cluster should now be ready to handle production workloads efficiently. If you encounter issues with larger clusters, consider configuring readiness probes to verify that your Elasticsearch Pods are ready to accept traffic.
How to Deploy Kibana on Kubernetes:
Deploying Kibana on Kubernetes allows you to visualize and manage your Elasticsearch data easily. Follow these steps to get Kibana up and running on your Kubernetes cluster.
Prerequisites
A running Elasticsearch cluster on Kubernetes.
Helm installed and configured.
Steps to Deploy Kibana
Add the Helm Chart for Kibana:
Use the official Helm chart to install Kibana. Run the following command, making sure to replace <CLIENT_SERVICE_NAME> with the name of your Elasticsearch client service. If you followed the 7-Pod Elasticsearch guide, the service name would be elasticsearch-client.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install --name elasticsearch --set \
name=elasticsearch,master.replicas=3,coordinating.service.type=LoadBalancer bitnami/elasticsearchVerify the Deployment:
After running the Helm command, check to ensure that Kibana is running properly by listing all resources:
[Output]
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-elasticsearch-coordinating-only-694b5f94f8-896k5 1/1 Running 0 3m55s
pod/elasticsearch-elasticsearch-coordinating-only-694b5f94f8-jvdrn 1/1 Running 0 3m55s
pod/elasticsearch-elasticsearch-data-0 1/1 Running 0 3m55s
pod/elasticsearch-elasticsearch-data-1 1/1 Running 0 3m27s
pod/elasticsearch-elasticsearch-master-0 1/1 Running 0 3m55s
pod/elasticsearch-elasticsearch-master-1 1/1 Running 0 3m35s
pod/elasticsearch-elasticsearch-master-2 1/1 Running 0 3m16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-elasticsearch-coordinating-only LoadBalancer 10.245.13.251 <Your_IP> 9200:32270/TCP 3m56s
service/elasticsearch-elasticsearch-discovery ClusterIP None <none> 9300/TCP 3m56s
service/elasticsearch-elasticsearch-master ClusterIP 10.245.0.78 <none> 9300/TCP 3m56s
service/kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 30m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/elasticsearch-elasticsearch-coordinating-only 2/2 2 2 3m55s
NAME DESIRED CURRENT READY AGE
replicaset.apps/elasticsearch-elasticsearch-coordinating-only-694b5f94f8 2 2 2 3m55s
NAME READY AGE
statefulset.apps/elasticsearch-elasticsearch-data 2/2 3m56s
statefulset.apps/elasticsearch-elasticsearch-master 3/3 3m56s Access Kibana:
Open your web browser and navigate to http://< YOUR KIBANA IP>:5601. You should see the Kibana interface, where you can start visualizing your Elasticsearch data.
.webp)
For more insights and updates, follow me on LinkedIn and Medium.
#Elasticsearch #Kubernetes #Helm #DevOps #DataVisualization #ContainerOrchestration #CloudComputing #Kibana #ClusterManagement #Scalability #Appnetwise #Greenbyte