Install Apache Kafka – Step-by-Step Guide for Beginners

💡 Introduction
Apache Kafka is a distributed event-streaming platform used for real-time data pipelines and stream processing.
It allows different systems to communicate through publish-subscribe messaging with high throughput, reliability, and scalability.

In this tutorial, we’ll learn how to install and run Kafka on your local machine (Windows, Linux, or macOS) using both manual and Docker approaches.

---

⚙️ 1️⃣ What You’ll Need

Before you start, ensure you have:

• ☕ Java 8+ (JDK) installed and added to PATH

• 🧰 Apache Kafka (latest stable release)

• 🐧 Linux, 🪟 Windows, or 🍎 macOS

• 🐋 (Optional) Docker for containerized setup

---

📦 2️⃣ Download and Extract Kafka

🔹 For Manual Installation

1. Go to the official Kafka downloads page:
   👉 https://kafka.apache.org/downloads

2. Choose a binary version (e.g., kafka_2.13-3.7.0.tgz) and extract it:

   tar -xzf kafka_2.13-3.7.0.tgz
   cd kafka_2.13-3.7.0
   

---

🧠 3️⃣ Start Zookeeper

Kafka requires Zookeeper to manage brokers and cluster metadata.

Run Zookeeper using the built-in script:

bin/zookeeper-server-start.sh config/zookeeper.properties

If you’re on Windows:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

📋 Tip:
By default, Zookeeper listens on port 2181.

---

📡 4️⃣ Start Kafka Broker

Open a new terminal and run:

bin/kafka-server-start.sh config/server.properties

On Windows:

.\bin\windows\kafka-server-start.bat .\config\server.properties

✅ You should see a message like:

INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

📋 Default Kafka ports:

Component	Port
Zookeeper	2181
Kafka Broker	9092

---

💬 5️⃣ Create a Kafka Topic

Let’s create a topic named jbpm-events:

bin/kafka-topics.sh --create \
  --topic jbpm-events \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 --partitions 1

✅ To list topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

---

✉️ 6️⃣ Send and Read Messages

📨 Producer (send messages)

bin/kafka-console-producer.sh --topic jbpm-events --bootstrap-server localhost:9092

Type your messages and press Enter after each.

📥 Consumer (read messages)

Open another terminal:

bin/kafka-console-consumer.sh --topic jbpm-events --from-beginning --bootstrap-server localhost:9092

You’ll see the same messages appear on the consumer terminal. 🎉

---

🐋 7️⃣ Optional: Run Kafka via Docker

For a faster setup, use Docker Compose.

Create a file docker-compose.yml:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

Run:

docker-compose up -d

✅ Kafka and Zookeeper will start automatically.

---

🧹 8️⃣ Stop and Clean Up

When done:

# Stop Kafka and Zookeeper
bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh

Or with Docker:

docker-compose down

---

🧠 9️⃣ Common Issues & Fixes

Issue	Cause	Fix
Address already in use	Ports 9092/2181 busy	Stop old services or change ports
Connection refused	Zookeeper not started	Start Zookeeper first
Broker not available	Misconfigured listeners	Update server.properties host settings

---

🧩 10️⃣ Next Steps

Once Kafka is running, you can:

• Integrate it with jBPM for event-driven workflows

• Use Spring Boot + Kafka for microservice messaging

• Monitor with Prometheus + Grafana

• Stream analytics using Kafka Streams or Flink

---

👉 Watch Install Apache Kafka in Action better:

Here's a quick video to help you understand Install Apache Kafka in Action better: Coming soon 

🏁 Conclusion

🎯 You’ve successfully installed Apache Kafka!
Now you can start sending and receiving messages, and integrate it with powerful platforms like jBPM, Camunda, or Spring Boot to build real-time data pipelines and process automation.

💼 Professional Support Available 

If you are facing issues in real projects related to enterprise backend development or workflow automation, I provide paid consulting, production debugging, project support, and focused trainings. 

Technologies covered include Java, Spring Boot, PL/SQL, Azure, and workflow automation (jBPM, Camunda BPM, RHPAM).

📧 Contact: ishikhanirankari@gmail.com | info@realtechnologiesindia.com

🌐 Website: IT Trainings | Digital metal podium