Why Centralized Logging

Picture a fleet of ships sailing across oceans. Each ship keeps its own logbook, but if the admiral wants to know what’s happening across the fleet, he can’t read every book individually. Kubernetes clusters face the same challenge: each pod generates logs, but without centralized aggregation, engineers can’t see the bigger picture.

The EFK Stack (Fluentd, Elasticsearch, Kibana) was born as the fleet’s central logbook, collecting, storing, and visualizing logs from across the cluster.

Fluentd – The Log Collector

• Definition: Fluentd is a data collector that gathers logs from pods and nodes.
• Key Features:
  • Flexible input/output plugins.
  • Parses and transforms logs.
  • Sends logs to multiple destinations (Elasticsearch, S3, Kafka).
• Analogy: Fluentd is like the messenger - collecting logbooks from ships and delivering them to headquarters.

Elasticsearch – The Log Warehouse

• Definition: Elasticsearch is a search and analytics engine that stores logs.
• Key Features:
  • Full‑text search.
  • Fast indexing and querying.
  • Scalable storage for millions of log entries.
• Analogy: Elasticsearch is like the warehouse - organizing logbooks so they can be searched instantly.

Kibana – The Log Dashboard

• Definition: Kibana is a visualization tool that connects to Elasticsearch.
• Key Features:
  • Interactive dashboards.
  • Powerful search and filtering.
  • Visualizations for trends, anomalies, and alerts.
• Analogy: Kibana is like the admiral’s command center - turning raw logs into actionable insights.

Global Context

• Enterprises: Use EFK for compliance, troubleshooting, and monitoring across thousands of pods.
• Cloud Providers: Managed Kubernetes services integrate EFK with cloud logging systems.
• Community: EFK is widely adopted as the de‑facto logging stack for Kubernetes.

Hands‑On Exercise

1. Access Kibana and search logs:
   • Filter logs by pod name.
   • Visualize error trends over time.
2. Reflect: How does Fluentd collect, Elasticsearch store, and Kibana visualize logs for cluster‑wide observability?

Deploy Kibana:

kubectl apply -f https://k8s.io/examples/kibana-deployment.yaml

Deploy Elasticsearch:

kubectl apply -f https://k8s.io/examples/elasticsearch-deployment.yaml

Deploy Fluentd as a DaemonSet:

kubectl apply -f https://k8s.io/examples/fluentd-daemonset.yaml

The Hacker’s Notebook

• Fluentd is the messenger - collecting logs from pods.
• Elasticsearch is the warehouse - storing logs for fast search.
• Kibana is the command center - visualizing logs for insights.
• Lesson for engineers: Don’t chase logs across pods - centralize them.
• Hacker’s mindset: Treat logs as your evidence. With EFK, you can trace issues across fleets of workloads.