Imagine running a fleet of delivery trucks. Without GPS trackers or fuel logs, you’d have no idea if trucks are stuck, wasting fuel, or breaking down. In production, monitoring and logging are those trackers - they give visibility into container health, performance, and behavior, ensuring smooth operations and quick troubleshooting.

Observability Foundations

1. Why Monitoring & Logging Matter

• Monitoring: Tracks metrics like CPU, memory, network usage, and application performance.
• Logging: Captures events and errors for debugging and auditing.
• Goal: Ensure reliability, detect problems early, and support compliance.

2. Monitoring Tools for Docker

• Prometheus: Collects metrics from containers and services.
• Grafana: Visualizes metrics with dashboards.
• cAdvisor: Monitors container resource usage.
• Docker Stats: Built‑in command for quick checks.

3. Logging Tools for Docker

• Docker Logging Drivers: Send logs to files, syslog, or external systems.
• ELK Stack (Elasticsearch, Logstash, Kibana): Centralized log collection and visualization.
• Fluentd: Flexible log aggregator.
• Cloud Logging Services: AWS CloudWatch, Azure Monitor, GCP Logging.

4. Best Practices

• Centralize logs for easier search and analysis.
• Use structured logging (JSON format).
• Set alerts for anomalies (high CPU, failed requests).
• Rotate logs to prevent disk exhaustion.
• Monitor both system metrics and application metrics.

Things to Remember

• Monitoring ensures proactive detection of issues.
• Logging provides detailed context for debugging.
• Tools like Prometheus + Grafana and ELK are industry standards.
• Best practices prevent blind spots and resource overload.

Hands‑On Lab

Step 1: Enable Docker Stats

docker stats

Shows CPU, memory, and network usage for running containers.

Step 2: Setup Prometheus + Grafana with Docker Compose

version: '3.8'
services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"

Step 3: Configure Logging Driver (JSON File Example)

docker run -d \
  --log-driver=json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  nginx

Step 4: ELK Stack Setup (Simplified Compose)

version: '3.8'
services:
  elasticsearch:
    image: elasticsearch:7.10.1
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
  kibana:
    image: kibana:7.10.1
    ports:
      - "5601:5601"

Practice Exercise

1. Run docker stats to monitor container resource usage.
2. Deploy Prometheus + Grafana with Docker Compose and create a dashboard.
3. Configure a logging driver for one container.
4. Set up ELK stack and visualize logs in Kibana.
5. Reflect on how monitoring and logging improve reliability and troubleshooting.

Visual Learning Model

Monitoring & Logging
   ├── Monitoring → Prometheus, Grafana, cAdvisor
   ├── Logging → ELK, Fluentd, drivers
   ├── Best Practices → centralize, structure, alerts
   └── Workflow → collect → visualize → act

The Hackers Notebook

Monitoring and logging are essential for production‑ready Docker applications. Monitoring tracks metrics, while logging captures events and errors. Tools like Prometheus, Grafana, and ELK provide visibility and insights. Best practices ensure proactive detection, efficient debugging, and reliable operations.