Kafka Performance Tuning Guide: Producer, Consumer & Broker Optimization

 

Apache Kafka is designed for high-throughput, low-latency event streaming, but installing Kafka and using the default configuration does not automatically guarantee optimal performance.

A Kafka cluster that performs well with a few thousand messages per second may behave very differently when traffic increases to hundreds of thousands or millions of events.

The key to Kafka performance tuning is understanding the complete data path:

Producer → Topic Partitions → Kafka Brokers → Consumer Groups

Performance problems can occur at any point in this pipeline.

In this practical guide, we will explore how to optimize:

• Kafka producers
• Kafka consumers
• Kafka brokers
• Topic partitions
• Message batching
• Compression
• Replication
• Consumer lag
• Disk and network utilization
• JVM and operating-system resources
• Kafka performance monitoring

The goal is not simply to make Kafka "faster." The real objective is to find the correct balance between throughput, latency, durability and resource utilization.

French version: https://shikhanirankari.blogspot.com/2026/08/optimisation-performance-kafka.html

🎥 Watch: Kafka Consumer Groups Explained

Prefer a visual explanation? Watch my video covering Kafka Consumer Groups, Partitions, Offsets and Rebalancing.

▶ Watch on YouTube: https://youtu.be/toR_ikiPG90

---

Kafka Performance Architecture

Before tuning individual parameters, consider the complete event flow.

Producer Applications

↓

Kafka Topic

↓

Partitions

↓

Kafka Brokers

↓

Consumer Group

↓

Downstream Applications

A producer sends records to topic partitions. Kafka brokers receive, persist and replicate those records. Consumers then fetch records from the partitions assigned to them.

This means poor Kafka performance may actually originate from:

• producer batching
• network bandwidth
• uneven partition distribution
• broker disk I/O
• replication overhead
• insufficient consumer parallelism
• slow downstream processing

For this reason, Kafka tuning should always be performed end-to-end rather than parameter-by-parameter.

---

1. Establish a Kafka Performance Baseline First

One of the biggest mistakes in Kafka tuning is changing configuration parameters before measuring current performance.

First establish a baseline.

Important metrics include:

Metric	What It Tells You
Producer throughput	How quickly producers publish data
Producer latency	Time required to acknowledge writes
Records/sec	Event processing rate
Bytes/sec	Network/data throughput
Consumer throughput	How quickly consumers process records
Consumer lag	How far consumers are behind producers
Broker CPU	Processing pressure
Disk utilization	Storage bottlenecks
Network utilization	Network saturation
Request latency	Broker/client response performance

Kafka includes performance-testing utilities such as:

kafka-producer-perf-test.sh

and:

kafka-consumer-perf-test.sh

Use performance tests before and after configuration changes.

The basic tuning process should be:

Measure → Identify Bottleneck → Tune → Test → Compare → Repeat

Never assume that a configuration copied from another Kafka environment will produce the same result in yours.

---

2. Kafka Producer Performance Tuning

The producer is the first place to investigate when Kafka write throughput is lower than expected.

Several producer properties have a significant impact on performance.

---

2.1 Tune batch.size

Kafka producers do not necessarily send every record individually.

Records going to the same partition can be grouped into batches.

batch.size=65536

Larger batches can:

• reduce the number of network requests
• improve compression
• increase throughput

However, excessively large batches can increase memory consumption and potentially affect latency.

For throughput-oriented workloads, test progressively larger batch sizes rather than immediately selecting an extremely large value.

---

2.2 Tune linger.ms

linger.ms controls how long the producer may wait for additional records before sending a batch.

Example:

linger.ms=10

A small delay can allow more records to enter each batch.

This creates an important trade-off:

Smaller linger.ms

→ lower waiting time
→ potentially smaller batches

Larger linger.ms

→ better batching
→ potentially higher throughput
→ potentially higher latency

For high-throughput systems, values such as 5–20 ms can be useful starting points for benchmarking, but the correct value depends on your workload.

---

2.3 Enable Compression

Compression can significantly reduce network traffic and broker storage requirements.

Example:

compression.type=lz4

Common options include:

none
gzip
snappy
lz4
zstd

For many high-throughput Kafka workloads, LZ4 provides a useful balance between compression efficiency and CPU usage.

Zstandard (zstd) can provide stronger compression and may be worth benchmarking when network bandwidth or storage is particularly important.

Do not select compression based only on compression ratio.

Measure:

• producer CPU
• broker CPU
• network bandwidth
• end-to-end latency
• storage utilization

---

2.4 Understand acks

The acks configuration controls producer durability.

acks=all

With acks=all, the producer waits for acknowledgements according to the partition's in-sync replica requirements.

This provides stronger durability than:

acks=1

where only the partition leader must acknowledge the record.

The decision should therefore be based on your business requirement, not simply on maximum benchmark throughput.

For business-critical event-driven applications, stronger durability is often more important than achieving the highest possible raw throughput.

---

2.5 Enable Idempotence Where Appropriate

For applications where duplicate writes caused by retries are unacceptable, Kafka producer idempotence is important.

enable.idempotence=true

Idempotence helps prevent duplicate records caused by producer retries while maintaining reliable delivery semantics.

For financial, workflow and enterprise integration events, reliability should normally be considered alongside throughput.

---

2.6 Producer Buffer Memory

The producer maintains records in memory while waiting to send them.

buffer.memory=67108864

If producers generate events faster than Kafka can accept them, insufficient buffering may cause blocking.

Increasing memory blindly is not the solution, however.

Persistent buffer pressure may indicate:

• slow brokers
• network problems
• insufficient partitions
• broker throttling
• downstream cluster saturation

Always investigate the root cause.

---

Recommended Producer Starting Configuration

For a throughput-oriented workload, a test configuration might look like:

bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092

acks=all

enable.idempotence=true

batch.size=65536

linger.ms=10

compression.type=lz4

buffer.memory=67108864

Treat these values as a benchmarking starting point, not universal production settings.

---

3. Kafka Consumer Performance Tuning

Producer optimization is only half of the Kafka performance equation.

A cluster can accept millions of events while consumers continuously fall behind.

The most important indicator is usually:

Consumer Lag

Consumer lag represents the difference between the latest offset available in a partition and the offset processed by the consumer.

Increasing lag indicates that consumers are not keeping up with incoming events.

---

3.1 Tune fetch.min.bytes

This configuration determines the minimum amount of data the broker attempts to return in a fetch response.

fetch.min.bytes=65536

Increasing it can reduce the number of small fetch requests and improve throughput.

However, waiting for more data can increase latency in low-volume workloads.

---

3.2 Tune fetch.max.wait.ms

This controls how long the broker may wait while attempting to satisfy fetch.min.bytes.

Example:

fetch.max.wait.ms=500

These two parameters should therefore be considered together:

fetch.min.bytes
fetch.max.wait.ms

For throughput-heavy systems, larger fetches can be more efficient than many small network requests.

---

3.3 Tune max.poll.records

This controls the maximum number of records returned by a call to poll().

Example:

max.poll.records=500

Increasing it may improve batch-processing efficiency.

But there is an important risk.

If processing each batch takes too long, the consumer may fail to poll Kafka within the expected interval.

This can result in:

Consumer Rebalance

Frequent rebalancing can significantly reduce Kafka consumer performance.

---

3.4 Tune max.poll.interval.ms

If records require substantial processing time, ensure that the consumer has enough time to process them.

max.poll.interval.ms=300000

Do not simply increase this value to hide slow processing.

Instead determine why processing is slow.

Possible causes include:

• database calls
• REST API calls
• external services
• CPU-intensive transformations
• synchronous processing
• slow storage
• insufficient application threads

---

4. Increase Consumer Parallelism

Kafka's scalability comes largely from partitions.

Within a consumer group:

One partition can be actively consumed by only one consumer at a time.

Suppose a topic has:

6 partitions

and the consumer group contains:

3 consumers

Kafka can distribute approximately two partitions to each consumer.

If the group contains six consumers, each consumer can potentially process one partition.

But if you create:

10 consumers

for only six partitions, four consumers cannot provide additional partition-level parallelism.

Therefore:

Maximum useful consumer parallelism ≈ number of partitions

This is why partition planning is a critical part of Kafka performance architecture.

---

5. Kafka Partition Optimization

Partitions provide parallelism for both producers and consumers.

More partitions can increase throughput because work can be distributed across multiple brokers.

However:

More partitions are not always better.

Every partition introduces overhead involving:

• metadata
• file handles
• replication
• leader management
• recovery
• memory
• network traffic

The correct partition count depends on:

• expected producer throughput
• consumer throughput
• number of consumers
• broker capacity
• message size
• retention
• replication factor
• future growth

Avoid creating thousands of partitions simply because Kafka supports them.

Partition count should be the result of capacity planning.

---

6. Avoid Kafka Hot Partitions

Even when a topic contains many partitions, poor key selection can create an uneven workload.

Imagine that most messages use the same key:

customerType=DEFAULT

If that key consistently maps to one partition, that partition can receive much more traffic than the others.

This creates a:

Hot Partition

Symptoms can include:

• one broker using significantly more CPU
• uneven disk utilization
• increased producer latency
• consumer lag on specific partitions

Choose partition keys that provide both:

1. required ordering
2. reasonable workload distribution

This is particularly important for high-volume event-driven microservices.

---

7. Kafka Broker Performance Tuning

When producers and consumers are configured correctly but performance remains poor, investigate the brokers.

Kafka broker performance depends heavily on:

CPU + Memory + Disk + Network

---

7.1 Disk Performance

Kafka continuously writes event data to disk.

Slow storage can therefore become a major bottleneck.

For high-throughput production workloads, fast SSD/NVMe storage generally provides better performance than slower disks.

Monitor:

• disk latency
• disk utilization
• read/write throughput
• I/O wait

Kafka's sequential I/O design is efficient, but the physical storage layer still matters.

---

7.2 Linux Page Cache

Kafka relies heavily on the operating system's page cache.

For this reason, allocating nearly all server RAM to the Kafka JVM can be counterproductive.

Memory must remain available for the operating system to cache frequently accessed log segments.

This is one reason Kafka broker sizing should not be approached like a conventional Java application where maximizing JVM heap is often assumed to be beneficial.

---

7.3 Network Capacity

Kafka can generate significant network traffic because data may travel through:

Producer → Broker Leader

then:

Leader → Replica Brokers

and later:

Broker → Consumers

For replicated topics, network traffic can therefore be considerably higher than the original producer traffic.

Monitor:

Network In
Network Out
Request Latency
Request Queue
Replication Traffic

A network bottleneck can limit Kafka even when CPU and disk utilization appear healthy.

---

8. Broker Thread Tuning

Two broker parameters sometimes investigated under heavy workloads are:

num.network.threads

and:

num.io.threads

Network threads process network requests while I/O threads handle request processing that can involve disk operations.

Increasing thread counts can help when metrics demonstrate that the existing thread pools are bottlenecks.

Do not increase them arbitrarily.

More threads can increase:

• CPU scheduling
• memory consumption
• context switching

Tune based on broker metrics and controlled performance tests.

---

9. Replication Factor vs Performance

Kafka replication provides fault tolerance.

A common production configuration is:

replication.factor=3

This means multiple copies of each partition are maintained across brokers.

Replication improves availability and durability but also consumes:

• disk capacity
• network bandwidth
• broker resources

Reducing replication solely to increase performance can create unacceptable reliability risks.

For enterprise production systems, reliability requirements should drive replication design.

---

10. Kafka Message Size Optimization

Large Kafka messages can cause performance problems.

Very large records increase:

• network transfer time
• memory requirements
• producer buffer pressure
• broker resource consumption
• consumer fetch requirements

Where possible, keep events compact.

Instead of placing a very large binary document directly into Kafka, consider storing the object in suitable object/content storage and publishing a reference through Kafka.

For example:

{
  "documentId": "DOC-10001",
  "location": "/documents/DOC-10001.pdf",
  "eventType": "DOCUMENT_CREATED"
}

This keeps the event stream lightweight while allowing downstream applications to retrieve the actual object when necessary.

---

11. Kafka Compression Strategy

Compression can improve Kafka performance when network bandwidth is a bottleneck.

The producer compresses record batches before transmitting them.

Conceptually:

100 MB Raw Events
        ↓
Compression
        ↓
Smaller Network Payload
        ↓
Kafka Broker

Better batching often also improves compression efficiency.

However, compression consumes CPU.

Therefore monitor both:

Network savings vs CPU cost

before selecting a compression algorithm.

---

12. Kafka Consumer Lag Optimization

Consumer lag is one of the most important Kafka operational metrics.

Suppose:

Latest Offset = 1,000,000
Consumer Offset = 850,000

Then:

Consumer Lag = 150,000

If lag continues increasing, the consumer cannot process events as quickly as producers generate them.

Possible solutions include:

• increase consumer instances
• increase partitions where appropriate
• optimize consumer processing
• batch database operations
• remove unnecessary synchronous calls
• optimize downstream APIs
• increase fetch efficiency
• investigate hot partitions

Do not automatically add consumers.

First identify whether the bottleneck is Kafka or the application processing the records.

---

13. Avoid Frequent Consumer Rebalancing

Consumer-group rebalancing temporarily disrupts normal partition processing.

Frequent rebalances may be caused by:

• slow processing
• unstable consumer instances
• inappropriate poll intervals
• network interruptions
• frequent deployments
• application crashes

Monitor consumer-group stability.

A healthy Kafka application should not continuously rebalance under normal operating conditions.

---

14. Throughput vs Latency: Understand the Trade-Off

There is no single Kafka configuration that provides maximum throughput, minimum latency and maximum durability simultaneously.

For example:

High Throughput

You may favor:

Larger batches
Higher linger
Compression
Larger fetches
More parallelism

Low Latency

You may favor:

Smaller linger
Smaller batches
Immediate processing
Carefully controlled fetch waiting

High Durability

You may favor:

acks=all
Replication
Idempotence
Appropriate minimum ISR

Kafka tuning is therefore a business trade-off, not simply a technical exercise.

---

15. Kafka Monitoring Metrics You Should Track

Kafka tuning without monitoring is guesswork.

Producer Metrics

Monitor:

record-send-rate
record-error-rate
request-latency-avg
request-latency-max
batch-size-avg
compression-rate-avg
buffer-available-bytes

Consumer Metrics

Monitor:

records-consumed-rate
records-lag-max
fetch-rate
fetch-latency-avg
bytes-consumed-rate

Broker Metrics

Monitor:

BytesInPerSec
BytesOutPerSec
MessagesInPerSec
UnderReplicatedPartitions
RequestQueueSize
NetworkProcessorAvgIdlePercent
RequestHandlerAvgIdlePercent

Also monitor infrastructure-level metrics:

CPU
Memory
Disk I/O
Disk Space
Network
JVM GC

Tools such as Prometheus and Grafana can provide useful dashboards for Kafka clusters and client applications.

---

16. Kafka Performance Tuning Example

Consider an e-commerce platform processing order events.

The initial producer configuration is:

batch.size=16384
linger.ms=0
compression.type=none
acks=all

The platform experiences high network usage while producers send many relatively small requests.

A benchmark configuration could be:

batch.size=65536
linger.ms=10
compression.type=lz4
acks=all
enable.idempotence=true

The team should then compare:

Before Tuning
-------------
Messages/sec
MB/sec
Average latency
p95 latency
p99 latency
CPU
Network utilization

After Tuning
------------
Messages/sec
MB/sec
Average latency
p95 latency
p99 latency
CPU
Network utilization

Only keep the change if the measured results improve the application's actual performance objectives.

---

17. Common Kafka Performance Tuning Mistakes

Avoid these common mistakes.

Mistake 1: Increasing Every Configuration Value

Higher values do not automatically mean better performance.

Mistake 2: Creating Too Many Partitions

Partitions provide parallelism but also consume resources.

Mistake 3: Ignoring Consumer Lag

Fast producers are useless if consumers continuously fall behind.

Mistake 4: Optimizing Only Kafka

Sometimes the real bottleneck is:

Database
REST API
Microservice
Network
Storage
Business logic

Mistake 5: Using Huge Kafka Messages

Kafka is an event-streaming platform, not a replacement for object or document storage.

Mistake 6: Sacrificing Reliability for Benchmark Numbers

A configuration that achieves the highest messages-per-second figure is not necessarily the correct production configuration.

Mistake 7: Tuning Without Benchmarking

Always measure before and after a change.

---

18. Kafka Performance Tuning Checklist

Before declaring a Kafka environment optimized, verify:

• Producer batching is tested
• linger.ms is appropriate for latency requirements
• Compression has been benchmarked
• acks matches durability requirements
• Idempotence is considered
• Partition distribution is balanced
• Partition count supports required parallelism
• Consumer lag is monitored
• Consumers process records efficiently
• Consumer rebalances are controlled
• Broker CPU is healthy
• Disk latency is acceptable
• Network capacity is sufficient
• Replication is healthy
• Under-replicated partitions are monitored
• Message sizes are controlled
• JVM and OS metrics are monitored
• Performance tests represent realistic production traffic

---

Kafka Performance Tuning Summary

Kafka performance optimization requires tuning the complete event-streaming pipeline rather than focusing on one configuration property.

A simplified tuning model is:

Producer
   ↓
Batch + Compress
   ↓
Partitions
   ↓
Kafka Brokers
   ↓
Disk + Network + Replication
   ↓
Consumer Group
   ↓
Parallel Processing

For producers, focus on:

batch.size + linger.ms + compression.type + acks + buffer.memory

For consumers, focus on:

fetch.min.bytes + fetch.max.wait.ms + max.poll.records + processing time + consumer lag

For brokers, focus on:

disk + network + partitions + replication + threads + page cache

Most importantly:

Do not tune Kafka based on assumptions. Benchmark your workload, identify the bottleneck, change one area at a time and measure the result.

A Kafka configuration that is perfect for a low-latency payment platform may be completely inappropriate for a high-throughput analytics pipeline.

---

Recommended Articles

Continue learning about Kafka, Spring Boot and event-driven architecture:

Event-Driven Microservices with Kafka & Spring Boot

Learn how Kafka enables asynchronous communication between Spring Boot microservices and how event-driven architectures reduce service coupling.

Kafka Consumer Groups Explained

Understand Kafka consumer groups, partitions, offsets, parallel processing and consumer rebalancing.

Spring Boot Microservices Explained

Learn the architecture and core building blocks behind enterprise Spring Boot microservices.

Kafka Event Streaming

Explore how Kafka provides scalable real-time event streaming between enterprise applications.

---

Conclusion

Apache Kafka is already designed for high performance, but production workloads require careful tuning.

The most effective Kafka optimization strategy is not to search for a "perfect Kafka configuration."

Instead:

Measure → Tune → Benchmark → Monitor → Repeat

By balancing producer batching, consumer parallelism, broker resources, partitions, compression and durability, Kafka can support highly scalable event-driven architectures while maintaining predictable latency and reliability.

If you are designing Kafka-based microservices, always perform performance testing with realistic message sizes, traffic patterns, partition counts and downstream processing before moving configuration changes into production.

📢 Need help with Java, workflows, or backend systems?

I help teams design scalable, high-performance, production-ready applications and solve critical real-world issues.

Services:

• Java & Spring Boot development
• Camunda Training / consulting
• Alfresco Training / consulting
• Workflow architecture guidance
• Workflow implementation (Camunda, Flowable – BPMN, DMN)
• Backend & API integrations (REST, microservices)
• Document management & ECM integrations (Alfresco)
• Performance optimization & production issue resolution

🔗 https://shikhanirankari.blogspot.com/p/professional-services.html

📩 Email: ishikhanirankari@gmail.com | info@realtechnologiesindia.com
🌐 https://realtechnologiesindia.com

✔ Available for quick consultations
✔ Response within 24 hours

🎥 Learn IT with Shikha on YouTube

Prefer learning through videos? Watch practical tutorials on Kafka, Camunda, Alfresco, Java, Spring Boot, Microservices and Enterprise Architecture.

▶ Subscribe to Learn IT with Shikha on YouTube