Kafka Deployment Architecture on Kubernetes: Scaling, HA & Monitoring
Running Apache Kafka on Kubernetes is much more than starting a few Kafka containers.
A production Kafka platform must address:
- broker availability;
- KRaft controller availability;
- persistent storage;
- pod and node failures;
- partition replication;
- broker scaling;
- resource allocation;
- network connectivity;
- observability;
- consumer lag;
- security;
- upgrades;
- disaster recovery.
Kafka is a distributed stateful platform, while Kubernetes dynamically schedules and replaces workloads. A successful architecture must therefore combine Kafka's replication and quorum model with Kubernetes orchestration and infrastructure primitives.
In this tutorial, we will explore a practical Kafka deployment architecture on Kubernetes, including scaling, high availability, KRaft, storage and monitoring.
French version: https://shikhanirankari.blogspot.com/2026/09/deploiement-kafka-kubernetes-scalabilite-ha-monitoring.html
1. Why Run Kafka on Kubernetes?
Kafka and Kubernetes solve different problems.
Apache Kafka provides distributed event streaming.
Kubernetes provides container orchestration.
Together, they can provide a platform for:
Microservices │ ▼ Event Producers │ ▼ Apache Kafka │ ▼ Event Consumers │ ▼ Business Applications
Kubernetes can help automate deployment, scheduling and infrastructure operations, while Kafka remains responsible for topics, partitions, replication and event durability.
A typical architecture may look like:
Kubernetes Cluster ┌──────────────────────────────────────────────┐ │ │ │ Kafka Platform │ │ │ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │ │Broker 1│ │Broker 2│ │Broker 3│ │ │ └────────┘ └────────┘ └────────┘ │ │ │ │ │ │ │ PVC PVC PVC │ │ │ │ KRaft Controller Quorum │ │ │ │ Monitoring + Metrics + Alerts │ │ │ └──────────────────────────────────────────────┘
The important point is that container orchestration does not replace Kafka's own high-availability mechanisms.
2. Kafka Is a Stateful Workload
Kafka brokers store partition data on disk.
For example:
Broker 1 ├── orders-0 ├── payments-1 └── customers-2 Broker 2 ├── orders-1 ├── payments-2 └── customers-0
This makes Kafka fundamentally different from a stateless REST API.
A stateless application pod can often disappear and be replaced without local application state.
Kafka requires careful handling of:
Broker identity Persistent storage Partition replicas Network identity Cluster metadata
Kubernetes StatefulSets are specifically designed for workloads that require persistent identity or storage: their pods maintain stable identifiers across scheduling operations.
When using an operator such as Strimzi, however, let the operator manage the underlying Kubernetes resources instead of manually managing generated workloads.
3. Modern Kafka Architecture: KRaft
Modern Kafka deployments use KRaft — Kafka Raft metadata mode rather than the older ZooKeeper-based architecture.
Conceptually:
KRaft Controller Quorum ┌─────┬─────┬─────┐ │ C1 │ C2 │ C3 │ └──┬──┴──┬──┴──┬──┘ │ │ │ ────────┴─────┴─────┴──────── │ Kafka Brokers ┌─────────┼─────────┐ ▼ ▼ ▼ Broker 1 Broker 2 Broker 3
Controllers maintain Kafka cluster metadata.
Brokers handle data traffic and partition storage.
For production environments, separating the conceptual responsibilities of controllers and brokers makes capacity planning easier.
Current Strimzi examples demonstrate separate KafkaNodePool resources with three controller nodes and three broker nodes.
4. Kafka Brokers Across Kubernetes Nodes
Imagine a Kubernetes cluster containing three worker nodes:
Kubernetes Cluster Worker Node 1 └── Kafka Broker 1 Worker Node 2 └── Kafka Broker 2 Worker Node 3 └── Kafka Broker 3
This is much better for availability than accidentally scheduling all brokers onto one Kubernetes worker.
Consider the following architecture:
Node 1 ├── Broker 1 └── PVC 1 Node 2 ├── Broker 2 └── PVC 2 Node 3 ├── Broker 3 └── PVC 3
If Node 2 fails, Brokers 1 and 3 may remain available.
This is why pod placement, anti-affinity and topology-aware scheduling are important parts of Kafka architecture.
5. Persistent Storage for Kafka
Kafka data should survive pod restarts.
Conceptually:
Kafka Broker Pod │ ▼ PersistentVolumeClaim │ ▼ Persistent Volume │ ▼ Storage Infrastructure
Without appropriate persistent storage, replacing a broker pod could also mean rebuilding its local partition data.
For production environments, storage design should consider:
- latency;
- throughput;
- IOPS;
- capacity;
- retention;
- failure domains;
- volume expansion;
- backup strategy;
- recovery time.
Do not select Kafka storage based only on capacity.
Disk performance directly affects Kafka throughput and latency.
6. Partition Replication Provides Kafka-Level HA
Suppose we create:
Topic: orders Partitions: 3 Replication Factor: 3
Kafka can distribute replicas approximately like this:
Broker 1 Broker 2 Broker 3 orders-0 L F F orders-1 F L F orders-2 F F L L = Leader F = Follower
If one broker becomes unavailable, another in-sync replica can potentially become leader.
This is the foundation of Kafka high availability.
7. Replication Factor and min.insync.replicas
A common durability pattern is:
replication.factor = 3 min.insync.replicas = 2 producer acks = all
This combination is important.
With replication factor 3, Kafka maintains three replicas for each partition.
With:
min.insync.replicas = 2
and producer:
acks=all
Kafka can require sufficient in-sync replicas before acknowledging a successful write.
Apache Kafka's current documentation explicitly gives replication factor 3, min.insync.replicas=2, and acks=all as a typical durability scenario.
This improves durability but also means availability and durability have to be balanced deliberately.
If the required number of in-sync replicas is unavailable, writes can fail rather than silently weakening the configured durability guarantee.
8. Kubernetes Availability Is Not Kafka Availability
This distinction is critical.
Kubernetes may report:
Pod = Running
but Kafka may still have:
Under-replicated partitions Offline partitions Controller problems Replica lag Storage pressure Consumer lag Request latency
Therefore:
Pod health alone is not enough to determine Kafka health.
You need observability at both levels:
Kubernetes Infrastructure + Kafka Cluster + Producer / Consumer Applications
9. Kubernetes Scheduling for Kafka
Kafka brokers should be distributed intelligently.
Useful Kubernetes concepts include:
Pod Anti-Affinity Node Affinity Topology Spread Taints and Tolerations Dedicated Nodes Availability Zones
The goal is to avoid:
Worker Node 1 ├── Broker 1 ├── Broker 2 └── Broker 3
because one worker-node failure could remove the entire broker layer.
Prefer distribution such as:
Worker Node 1 → Broker 1 Worker Node 2 → Broker 2 Worker Node 3 → Broker 3
For multi-zone infrastructure:
Zone A → Broker 1 Zone B → Broker 2 Zone C → Broker 3
The exact design depends on your cloud, storage and failure-domain architecture.
10. Scaling Kafka on Kubernetes
Kafka scaling is not equivalent to scaling a stateless web service.
For a stateless application:
3 Pods → 6 Pods
may immediately increase available compute capacity.
For Kafka:
3 Brokers → 6 Brokers
adds brokers, but existing partitions do not automatically become perfectly balanced merely because more broker processes exist.
Scaling therefore involves two related concerns:
Broker Capacity + Partition Distribution
After adding capacity, partition replicas may need to be reassigned or rebalanced so that the new brokers actually receive useful workload.
11. Horizontal Broker Scaling
Suppose the cluster initially contains:
Broker 1 Broker 2 Broker 3
As traffic grows:
Producer throughput ↑ Storage usage ↑ Partition count ↑ Consumer traffic ↑
you may add:
Broker 4 Broker 5 Broker 6
But scaling should be driven by metrics and capacity planning rather than simply CPU percentage.
Useful signals include:
Disk utilization Disk throughput Network throughput Request latency Partition count Replica distribution CPU Memory Page cache Replication health
12. Partition Count and Scaling
Kafka parallelism is heavily influenced by partitions.
For example:
orders topic Partitions = 12
A consumer group may process those partitions across multiple consumers:
Consumer Group Consumer 1 → P0, P1, P2 Consumer 2 → P3, P4, P5 Consumer 3 → P6, P7, P8 Consumer 4 → P9, P10, P11
Adding consumers beyond the available partitions does not create unlimited parallelism.
Therefore, partition planning should consider:
- expected throughput;
- consumer parallelism;
- ordering requirements;
- future growth;
- broker count;
- recovery time.
Avoid creating huge partition counts without a capacity reason.
13. Scaling KRaft Controllers
KRaft controller scaling is different from broker scaling.
The controller layer forms a metadata quorum.
Current Strimzi documentation distinguishes static and dynamic KRaft controller quorums. With static controller quorums, scaling requires downtime; dynamic controller quorums allow controllers to be added or removed without downtime through quorum membership changes.
That means you should not treat controller replicas like ordinary stateless pods.
A production architecture should explicitly plan:
Controller count Controller placement Failure domains Metadata storage Quorum availability
14. Using Strimzi for Kafka on Kubernetes
One popular way to manage Kafka on Kubernetes is the Strimzi project.
Strimzi provides Kubernetes operators and custom resources for managing Kafka deployments. Its documentation describes deployment across different Kubernetes distributions and configuration through custom resources.
Instead of manually managing every Kafka pod and Kubernetes object, the architecture becomes:
Kafka Custom Resource │ ▼ Strimzi Cluster Operator │ ▼ Kafka Infrastructure │ ├── Controllers ├── Brokers ├── Services ├── Storage └── Supporting Resources
This provides a Kubernetes-native operational model for Kafka.
15. Example Kafka Node Pool Architecture
Conceptually, a production deployment might contain:
Kafka Cluster Controller Node Pool ├── Controller 1 ├── Controller 2 └── Controller 3 Broker Node Pool ├── Broker 1 ├── Broker 2 └── Broker 3
Current Strimzi examples use separate controller and broker KafkaNodePool resources and persistent claims for storage.
A simplified conceptual configuration looks like:
apiVersion: kafka.strimzi.io/v1 kind: KafkaNodePool metadata: name: broker spec: replicas: 3 roles: - broker storage: type: persistent-claim size: 100Gi
The exact schema depends on the Strimzi version in use, so production manifests should always be validated against the matching operator documentation.
16. Kafka Resource Planning
Avoid treating Kafka resource requests as arbitrary values.
Consider:
CPU Memory Heap Page Cache Disk Network Partition Count Message Size Traffic Pattern Retention Replication
Kafka performance depends heavily on operating-system page cache.
Therefore, allocating nearly all container memory to JVM heap can be counterproductive.
Conceptually:
Pod Memory │ ├── JVM Heap │ └── OS / Page Cache
Capacity testing should use workloads that resemble production traffic.
17. Why Monitoring Kafka on Kubernetes Is Essential
A Kafka cluster can appear healthy from Kubernetes while applications are already experiencing problems.
You therefore need to monitor several layers:
Layer 1 → Kubernetes Layer 2 → Kafka Brokers Layer 3 → Topics & Partitions Layer 4 → Producers Layer 5 → Consumers Layer 6 → Business Events
This layered monitoring model helps answer different questions.
Kubernetes: Is the infrastructure healthy?
Kafka: Is the cluster healthy?
Consumers: Are applications keeping up?
Business: Are expected events actually flowing?
18. Kafka Metrics to Monitor
Important Kafka metrics include:
Broker health
Broker availability Request rate Request latency Network throughput CPU Memory Disk usage
Replication
Under-replicated partitions Offline partitions ISR changes Replica lag Leader distribution
Producers
Record send rate Request latency Error rate Retries
Consumers
Consumer lag Consumption rate Commit rate Rebalances Errors
Storage
Disk utilization Log growth Retention Disk latency Volume capacity
These metrics provide a much better picture than checking whether pods are merely running.
19. Prometheus and Grafana Monitoring Architecture
A common monitoring architecture is:
Kafka Brokers │ │ Metrics ▼ Metrics Exporter │ ▼ Prometheus │ ▼ Grafana │ ▼ Dashboards + Alerts
Strimzi's current example configuration demonstrates jmxPrometheusExporter metrics configuration as well as a Kafka Exporter configuration for topic and consumer-group visibility.
This architecture can provide dashboards for:
Broker Health Partition Replication Consumer Lag Throughput Latency Storage Resource Usage
20. Consumer Lag Is a Critical Application Metric
Consumer lag tells you how far a consumer group is behind the latest records in Kafka.
Conceptually:
Latest Kafka Offset = 1,000,000 Consumer Offset = 970,000 Lag = 30,000
Growing lag may indicate:
Slow consumer Insufficient consumers Downstream database latency Application errors Large traffic spike Network problems Partition imbalance
But lag should be interpreted together with traffic rate.
A lag of 30,000 events might be harmless for a consumer processing hundreds of thousands of events per second, but serious for another workload.
21. Monitor Under-Replicated Partitions
Suppose:
Replication Factor = 3
but one replica has fallen behind.
The partition may temporarily have fewer healthy in-sync copies than intended.
Under-replicated partitions can indicate:
Broker failure Slow disk Network issue Overloaded broker Replica recovery Resource pressure
Persistent replication problems should be investigated quickly.
22. Monitor Offline Partitions
Offline partitions are more severe.
If Kafka cannot find an eligible replica to lead a partition, that partition may become unavailable.
This can affect:
Producer writes Consumer reads Application availability
Production alerting should therefore distinguish between:
Warning: Under-replicated partitions Critical: Offline partitions
23. Monitor Disk Before It Becomes an Incident
Kafka is storage-intensive.
Monitor:
PVC utilization Filesystem utilization Disk latency Disk throughput Retention growth Log segment growth
Do not wait until:
Disk Usage = 99%
before acting.
A useful capacity-management process estimates:
Daily ingest × Retention period × Replication factor + Operational headroom
Compression and actual workload patterns also influence storage requirements.
24. Kafka Kubernetes Failure Scenario
Consider:
Node A → Broker 1 Node B → Broker 2 Node C → Broker 3
A worker node fails:
Node B ❌ Broker 2 ❌
Kafka's remaining in-sync replicas may continue serving partitions through leader election, depending on replication state and configuration.
Kubernetes can work to restore the workload.
The complete recovery path therefore involves two systems:
Kafka → Maintains partition availability Kubernetes → Restores infrastructure workload
This illustrates why Kafka HA and Kubernetes HA must be designed together.
25. PodDisruptionBudget and Planned Maintenance
Planned Kubernetes operations can evict pods.
For a Kafka cluster, you generally do not want multiple critical brokers to disappear simultaneously because of routine maintenance.
A PodDisruptionBudget can help Kubernetes understand voluntary disruption limits.
But remember:
A PodDisruptionBudget is not a replacement for Kafka replication.
It is an infrastructure safeguard that complements Kafka's own availability design.
26. Multi-Zone Kafka Architecture
For stronger resilience, brokers can be distributed across availability zones when the underlying infrastructure supports it.
Example:
Kubernetes Region │ ├── Zone A │ └── Broker 1 │ ├── Zone B │ └── Broker 2 │ └── Zone C └── Broker 3
Partition replicas should then be distributed so that one infrastructure failure does not remove every replica.
However, multi-zone Kafka also introduces:
Cross-zone latency Cross-zone traffic cost Storage constraints Network dependencies
Availability design must therefore consider both resilience and operational cost.
27. Kafka Networking on Kubernetes
Kafka networking is more complex than exposing a normal HTTP service because clients need broker connectivity.
You may have:
Internal applications External applications Cross-cluster clients Administrative tools Monitoring systems
Listeners should be designed according to these access patterns.
Conceptually:
Internal Microservices │ ▼ Internal Kafka Listener │ ▼ Kafka Brokers External Client │ ▼ External Listener │ ▼ Kafka Brokers
Avoid exposing Kafka externally unless the business architecture requires it.
28. Kafka Security on Kubernetes
Production Kafka architecture should also include:
TLS Encryption Authentication Authorization Secrets Management Network Policies Certificate Management Least Privilege Audit & Governance
Security should exist across:
Client → Kafka Broker → Broker Operations → Kafka Kubernetes → Workloads
For a deeper security discussion, link this section to your existing Kafka Security Best Practices: SSL, SASL, ACLs & Enterprise Governance article.
29. Production Deployment Architecture
A more complete production architecture looks like:
PRODUCERS │ ▼ Kafka Listeners │ ┌──────────────┴──────────────┐ │ Kubernetes Cluster │ │ │ │ KRaft Controller Quorum │ │ C1 C2 C3 │ │ │ │ Kafka Broker Node Pool │ │ │ │ B1 B2 B3 │ │ │ │ │ │ │ PVC PVC PVC │ │ │ │ Partition Replication │ │ │ └──────────────┬──────────────┘ │ ▼ CONSUMERS OBSERVABILITY │ ┌──────────┴──────────┐ │ │ Metrics Logs/Events │ ▼ Prometheus │ ▼ Grafana │ ▼ Alerts / SRE
This separates the main operational concerns:
Compute Storage Metadata quorum Data replication Networking Monitoring Applications
30. Kafka Scaling Strategy
A good scaling strategy evaluates:
| Signal | Possible Action |
|---|---|
| High CPU | Investigate workload and broker capacity |
| High disk usage | Expand capacity or adjust retention |
| High network throughput | Add capacity / review distribution |
| Uneven partitions | Rebalance replicas |
| Growing consumer lag | Scale or optimize consumers |
| High request latency | Investigate brokers, storage and network |
| High partition density | Add broker capacity or redesign |
| Storage latency | Improve storage architecture |
Do not use a single metric to determine scaling.
Kafka bottlenecks can move between CPU, disk, network, partitions and application consumers.
31. Kafka High Availability Checklist
Before calling a Kafka Kubernetes deployment production-ready, verify:
| Area | Check |
|---|---|
| Brokers | Multiple brokers deployed |
| Controllers | Resilient KRaft quorum |
| Placement | Brokers spread across failure domains |
| Storage | Persistent storage configured |
| Replication | Critical topics have appropriate replication |
| ISR | min.insync.replicas designed intentionally |
| Producers | Appropriate acknowledgements configured |
| Monitoring | Broker metrics available |
| Consumer Lag | Consumer groups monitored |
| Storage | Disk utilization and latency monitored |
| Alerts | Critical Kafka conditions alert operations |
| Security | TLS/authentication/authorization considered |
| Resources | CPU/memory/storage sized from workload |
| Scaling | Broker and partition scaling process defined |
| Maintenance | Planned disruptions controlled |
| Recovery | Broker/node failure tested |
| DR | Recovery requirements documented |
32. Common Kafka-on-Kubernetes Mistakes
Mistake 1 — Treating Kafka like a stateless application
Kafka stores distributed state.
Mistake 2 — Putting all brokers on one worker node
One infrastructure failure can have excessive impact.
Mistake 3 — Using ephemeral storage for production data
Kafka brokers need an intentional persistence strategy.
Mistake 4 — Scaling brokers without considering partitions
Additional brokers do not automatically solve every bottleneck.
Mistake 5 — Monitoring only Kubernetes pods
Running does not necessarily mean Kafka is healthy.
Mistake 6 — Ignoring consumer lag
The cluster can be healthy while applications are falling behind.
Mistake 7 — Ignoring disk latency
Kafka performance is heavily influenced by storage.
Mistake 8 — Weak replication settings
Replication and ISR configuration directly affect durability and availability.
Mistake 9 — No failure testing
HA should be tested, not assumed.
Mistake 10 — No capacity planning
Retention, replication and ingest growth can consume storage rapidly.
33. Recommended Production Design Principles
For a production Kafka deployment on Kubernetes, remember:
Kafka HA ≠ Only Kubernetes HA
Instead:
Production Kafka Reliability = Kafka Replication + KRaft Quorum + Persistent Storage + Failure-Domain Distribution + Resource Planning + Monitoring + Operational Automation
A Kubernetes operator can simplify operations, but architecture decisions still matter.
Final Thoughts
Running Apache Kafka on Kubernetes can provide a powerful platform for event-driven applications, but the deployment must respect Kafka's stateful and distributed architecture.
The core architecture can be summarized as:
Applications ↓ Producers ↓ Kafka Listeners ↓ Kafka Brokers ↕ Partition Replication ↕ Persistent Storage ↑ KRaft Controllers ↓ Consumers + Monitoring Prometheus Grafana Alerts
For high availability, focus on:
replication + quorum + failure-domain distribution
For scaling, focus on:
brokers + partitions + storage + workload distribution
For monitoring, focus on:
Kafka health + Kubernetes health + consumer health
And remember:
Kubernetes can restart a Kafka broker, but Kafka's own replication architecture is what protects partition availability and event durability.
A successful production design therefore combines the strengths of both platforms rather than expecting one to replace the responsibilities of the other.
Recommended Articles
I recommend adding these immediately after Final Thoughts to strengthen your Kafka topic cluster:
1. Kafka Security Best Practices: SSL, SASL, ACLs & Enterprise Governance
Read the Kafka Security guide
2. Kafka Consumer Groups Explained: Partitions, Offsets & Rebalancing
Read the Kafka Consumer Groups guide
3. Event-Driven Microservices with Kafka & Spring Boot
Add your existing published Blogger URL here rather than creating a new URL.
4. Kafka Architecture: Producers, Consumers, Brokers, Topics & Partitions
Add the existing published URL from your blog.
5. Kafka Performance Tuning: Producers, Consumers & Brokers
Add your existing published URL here.
This internal-link structure creates a useful topic path:
Kafka Architecture ↓ Kafka Consumer Groups ↓ Kafka Kubernetes Deployment ↓ Kafka Security ↓ Kafka Performance
🎥 Learn IT with Shikha on YouTube
Prefer learning through videos?
Watch practical tutorials on Apache Kafka, Spring Boot, Microservices, Camunda, Alfresco, Java and Enterprise Architecture.
Subscribe to Learn IT with Shikha on YouTube
You can also embed your relevant Kafka / Spring Boot / Event-Driven Microservices YouTube video above this section.
📢 Need help with Java, workflows, or backend systems?
I help teams design scalable, high-performance, production-ready applications and solve critical real-world issues.
Services:
- Java & Spring Boot development
- Camunda Training / consulting
- Alfresco Training / consulting
- Workflow architecture guidance
- Workflow implementation (Camunda, Flowable – BPMN, DMN)
- Backend & API integrations (REST, microservices)
- Document management & ECM integrations (Alfresco)
- Performance optimization & production issue resolution
🔗 https://shikhanirankari.blogspot.com/p/professional-services.html
📩 Email: ishikhanirankari@gmail.com | info@realtechnologiesindia.com
🌐 https://realtechnologiesindia.com
✔ Available for quick consultations
✔ Response within 24 hours
🎥 Learn IT with Shikha on YouTube
Prefer learning through videos? Watch practical tutorials on Kafka, Camunda, Alfresco, Java, Spring Boot, Microservices and Enterprise Architecture.▶ Subscribe to Learn IT with Shikha on YouTube
Comments
Post a Comment