Event-Driven Microservices with Kafka & Spring Boot: Async Enterprise Integration Guide
Modern enterprise applications are increasingly moving away from tightly coupled synchronous integrations toward event-driven microservices architecture.
Instead of one microservice calling another service and waiting for a response, services can publish business events such as:
OrderCreated PaymentCompleted CustomerRegistered InventoryUpdated ApplicationApproved DocumentUploaded
Other services can consume these events asynchronously and perform their own business operations.
One of the most widely used technologies for implementing this architecture is Apache Kafka, while Spring Boot with Spring for Apache Kafka makes Kafka integration straightforward for Java-based enterprise applications.
In this tutorial, we will understand how Kafka and Spring Boot work together to build event-driven microservices, including producers, consumers, topics, partitions, consumer groups, retries, error handling, and practical enterprise design patterns.
French version: https://shikhanirankari.blogspot.com/2026/08/microservices-evenementiels-kafka-spring-boot.html
🎥 Watch: Kafka Consumer Groups Explained
Prefer a visual explanation? Watch my video covering Kafka Consumer Groups, Partitions, Offsets and Rebalancing.
▶ Watch on YouTube: https://youtu.be/toR_ikiPG90
What Is Event-Driven Architecture?
In a traditional synchronous microservices architecture, one service often communicates directly with another using REST APIs.
For example:
Order Service | | REST v Inventory Service | | REST v Notification Service
This design works well for many applications, but as the number of services increases, direct dependencies between services can become difficult to manage.
In an event-driven architecture, services communicate through events instead.
Order Service | | OrderCreated Event v Apache Kafka | +------------------+ | | v v Inventory Service Notification Service
The Order Service does not need to know how Inventory Service or Notification Service processes the event.
It simply publishes the event.
The other services independently react to it.
This creates loose coupling between microservices.
Event-Driven Microservices Architecture with Kafka
Kafka Spring Boot Event Driven Microservices Architecture
Apache Kafka acts as the event backbone between microservices.
A typical architecture might contain:
Order Service | | publish v +-----------------------+ | Apache Kafka | | | | Topic: order-events | | P0 P1 P2 | +-----------------------+ | | | consume | consume v v Inventory Notification Service Service
The services that generate events are called producers, while applications that subscribe to and process those events are consumers.
Kafka stores events inside topics, and topics can be divided into partitions for scalability. Events using the same key can be routed to the same partition, which allows ordering to be preserved within that topic-partition. Apache Kafka officially documents producers and consumers as decoupled applications and topics as durable event stores.
Why Use Kafka for Microservices?
Kafka is particularly useful when enterprise applications need:
- asynchronous communication
- high-throughput messaging
- independent microservice scaling
- event replay
- distributed processing
- durable event storage
- multiple consumers for the same business event
- reduced dependencies between services
Consider an e-commerce application.
When an order is created, several activities may need to happen.
Order Created | +--> Update Inventory | +--> Process Analytics | +--> Send Notification | +--> Update Customer History
Without an event platform, the Order Service may need to call every downstream service.
With Kafka, the Order Service can publish one event:
OrderCreated
Multiple applications can independently consume it.
Kafka Core Concepts for Microservices
Before implementing Spring Boot services, let's understand the most important Kafka concepts.
1. Event
An event represents something that happened in the system.
Example:
{ "eventId": "EVT-10001", "eventType": "OrderCreated", "orderId": "ORD-501", "customerId": "CUST-200", "amount": 1250.00, "timestamp": "2026-08-10T10:30:00Z" }
Instead of sending commands such as:
UpdateInventory
an event-driven system often communicates a fact:
OrderCreated
Consumers decide what they need to do with that information.
2. Kafka Producer
A producer publishes events to Kafka.
For example:
Order Service | | OrderCreated v Kafka
In a Spring Boot application, KafkaTemplate provides a convenient abstraction for publishing Kafka records. Spring Boot can auto-configure KafkaTemplate when Kafka dependencies and configuration are present.
Example:
@Service public class OrderEventProducer { private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate; public OrderEventProducer( KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public void publishOrderCreated(OrderCreatedEvent event) { kafkaTemplate.send( "order-created", event.getOrderId(), event ); } }
Here:
order-created
is the Kafka topic.
And:
event.getOrderId()
is used as the Kafka message key.
3. Kafka Topic
A Kafka topic is a logical stream where related events are stored.
Examples:
order-created payment-completed inventory-updated customer-created application-approved
Multiple producers can publish events to a topic, and multiple consumers can subscribe to it.
Kafka does not automatically remove an event simply because one consumer has read it. Events are retained according to the topic's configured retention policy.
This is one of the important differences between Kafka and many traditional queue-based messaging architectures.
4. Kafka Partitions
Kafka topics are divided into partitions.
For example:
order-events Partition 0 Partition 1 Partition 2
Partitions allow Kafka workloads to be distributed across brokers and enable parallel processing.
Imagine these orders:
Order-101 Order-102 Order-103 Order-104
Kafka could distribute them across different partitions.
A key design point is that Kafka guarantees ordering within an individual topic-partition, rather than guaranteeing one global ordering across every partition.
This is why selecting the correct Kafka message key is extremely important.
5. Kafka Consumer
A consumer reads and processes Kafka events.
Spring Kafka provides the @KafkaListener annotation for implementing message-driven consumers.
Example:
@Service public class InventoryEventConsumer { @KafkaListener( topics = "order-created", groupId = "inventory-service" ) public void consume(OrderCreatedEvent event) { System.out.println( "Processing inventory for order: " + event.getOrderId() ); // Update inventory } }
Whenever a new event arrives on:
order-created
the consumer can process it asynchronously.
6. Kafka Consumer Groups
Consumer groups allow Kafka applications to process events in parallel.
Suppose a topic has three partitions:
order-events P0 P1 P2
And the microservice has three consumer instances:
Consumer 1 Consumer 2 Consumer 3
Kafka can distribute partitions across those consumers.
P0 ---> Consumer 1 P1 ---> Consumer 2 P2 ---> Consumer 3
Within the same consumer group, partitions are distributed among the participating consumers so that processing can be parallelized.
For a detailed explanation of partitions, offsets, consumer groups and rebalancing, also read:
Kafka Consumer Group Architecture Explained – Partitions, Offsets & Rebalancing
https://shikhanirankari.blogspot.com/2026/08/architecture-consumer-groups-kafka-partitions-offsets-rebalancing.html
Building Event-Driven Microservices with Spring Boot
Now let's implement a simple architecture.
We will create:
Order Service | | OrderCreatedEvent v Apache Kafka | +----------------------+ | | v v Inventory Service Notification Service
Step 1: Add Spring Kafka Dependency
For a Maven-based Spring Boot application, include Spring Kafka:
<dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>
When using Spring Boot dependency management, normally let Spring Boot manage the compatible Spring Kafka version rather than manually forcing an arbitrary version.
Step 2: Configure Kafka
Add the Kafka configuration to:
application.yml
Example:
spring: kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer consumer: group-id: order-service auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: spring.json.trusted.packages: "*"
Spring Boot exposes Kafka configuration primarily through:
spring.kafka.*
properties.
Step 3: Create the Event Object
Create:
public class OrderCreatedEvent { private String eventId; private String orderId; private String customerId; private Double amount; private String timestamp; public OrderCreatedEvent() { } public OrderCreatedEvent( String eventId, String orderId, String customerId, Double amount, String timestamp) { this.eventId = eventId; this.orderId = orderId; this.customerId = customerId; this.amount = amount; this.timestamp = timestamp; } // getters and setters }
A production event normally contains enough information for consumers to understand what happened without becoming unnecessarily dependent on the producer's internal implementation.
Step 4: Create Kafka Producer
Now create the producer:
@Service public class OrderProducer { private static final String TOPIC = "order-created"; private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate; public OrderProducer( KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public void publish(OrderCreatedEvent event) { kafkaTemplate.send( TOPIC, event.getOrderId(), event ); } }
Flow:
Order Service | | KafkaTemplate.send() v order-created Topic
Step 5: Create REST API
Let's expose an endpoint for creating an order.
@RestController @RequestMapping("/orders") public class OrderController { private final OrderProducer orderProducer; public OrderController(OrderProducer orderProducer) { this.orderProducer = orderProducer; } @PostMapping public ResponseEntity<String> createOrder( @RequestBody OrderCreatedEvent event) { orderProducer.publish(event); return ResponseEntity.ok( "Order event published successfully" ); } }
Example request:
POST /orders Content-Type: application/json
Payload:
{ "eventId": "EVT-10001", "orderId": "ORD-501", "customerId": "CUST-200", "amount": 1250, "timestamp": "2026-08-10T10:30:00Z" }
Step 6: Kafka Receives the Event
The execution flow now becomes:
Client | | POST /orders v Order Service | | KafkaTemplate v Kafka Topic order-created | +-------------------------+ | | v v Inventory Consumer Notification Consumer
The producer does not need to wait for both downstream business processes to finish before those consumers perform their work.
That is the core advantage of asynchronous enterprise integration.
Step 7: Create Inventory Consumer
@Service public class InventoryConsumer { @KafkaListener( topics = "order-created", groupId = "inventory-service" ) public void consume(OrderCreatedEvent event) { System.out.println( "Updating inventory for Order: " + event.getOrderId() ); // inventory update logic } }
Step 8: Create Notification Consumer
Another independent microservice can consume exactly the same business event.
@Service public class NotificationConsumer { @KafkaListener( topics = "order-created", groupId = "notification-service" ) public void consume(OrderCreatedEvent event) { System.out.println( "Sending notification for Order: " + event.getOrderId() ); // notification logic } }
Notice that the group IDs are different:
inventory-service notification-service
That matters because Inventory Service and Notification Service represent two independent logical subscribers.
Each needs to process the event.
Complete Event Flow
The complete process now looks like this:
1. Customer creates an order | v 2. Order Service creates OrderCreatedEvent | v 3. KafkaTemplate publishes event | v 4. Kafka stores the event in order-created topic | +-----+------+ | | v v 5. Inventory 6. Notification Service Service | | v v Update Stock Send Email/SMS
The services can evolve and scale independently.
Synchronous REST vs Event-Driven Kafka
REST-based Integration
Service A | v Service B | v Service C
Service A may depend on Service B being available.
Service B may depend on Service C.
Failures can therefore propagate through a synchronous call chain unless carefully controlled.
Kafka Event-Driven Integration
Service A | v Kafka / \ v v B C
The producer and consumers are much more loosely coupled.
Kafka itself is the communication backbone.
Event-Driven Architecture Does Not Mean “No REST”
This is an important architectural point.
REST and Kafka solve different integration problems.
Use REST APIs when you need an immediate request-response interaction.
For example:
GET /customers/501
The caller wants a customer response immediately.
Kafka is often more appropriate when communicating that something has happened:
CustomerCreated OrderCompleted PaymentReceived DocumentProcessed
Modern enterprise architectures frequently use both.
Synchronous Queries ---> REST Business Events ---> Kafka
Event Naming Best Practices
Events should normally describe something that has already happened.
Good examples:
OrderCreated PaymentCompleted LoanApproved CustomerRegistered DocumentUploaded
Less ideal event naming:
DoPayment UpdateCustomer CallInventory
Those names resemble commands rather than events.
Event names based on completed business facts make an event-driven architecture easier to understand.
Kafka Message Key Strategy
The Kafka message key has an important architectural role.
Consider:
kafkaTemplate.send( "order-created", event.getOrderId(), event );
Here:
Order ID = Kafka Key
Events with the same key can be routed consistently to the same partition.
Example:
Order-100 CREATED Order-100 PAID Order-100 SHIPPED
Using:
Order-100
as the key helps keep events for that order together within the same partition, where Kafka preserves ordering.
Event Schema Design
Avoid exposing an internal database entity directly as your Kafka contract.
Instead of publishing:
OrderEntity
prefer a dedicated event schema:
OrderCreatedEvent
Example:
{ "eventId": "35e8c2", "eventType": "OrderCreated", "eventVersion": "1.0", "timestamp": "2026-08-10T10:30:00Z", "data": { "orderId": "ORD-501", "customerId": "CUST-200", "amount": 1250 } }
This gives you greater freedom to evolve internal application models without immediately changing your public event contract.
Event Versioning
Enterprise systems evolve.
Suppose version 1 contains:
{ "orderId": "ORD-501", "amount": 1250 }
Later you may add:
{ "orderId": "ORD-501", "amount": 1250, "currency": "INR" }
Event consumers should be designed with compatibility in mind.
Common strategies include:
Schema evolution Backward-compatible fields Event version metadata Schema Registry where appropriate
Handling Kafka Consumer Failures
What happens if the consumer cannot process an event?
For example:
OrderCreated | v Inventory Service | X Database unavailable
Simply retrying forever may create operational problems.
A production architecture should define an explicit failure policy.
Typical strategy:
Kafka Event | v Consumer | X Failure | v Retry | X Failure | v Dead Letter Topic
For example:
order-created order-created-retry order-created-dlt
Spring Kafka provides support for retry-related patterns, including retryable topics.
Idempotency Is Critical
Distributed systems can encounter duplicate delivery or repeated processing scenarios.
Therefore, consumers should often be designed to be idempotent.
Suppose this event arrives twice:
{ "eventId": "EVT-10001", "orderId": "ORD-501" }
The consumer should avoid accidentally performing the same irreversible business operation twice.
One approach is keeping processed event IDs.
eventId ---------------- EVT-10001 EVT-10002 EVT-10003
Before executing:
Has EVT-10001 already been processed? YES -> Ignore NO -> Process and record eventId
This becomes particularly important for operations involving payments, financial transactions or external integrations.
The Dual-Write Problem
Consider:
saveOrderToDatabase(); kafkaTemplate.send("order-created", event);
What happens if:
Database write succeeds
but:
Kafka publish fails?
The database says an order exists, while downstream applications never receive the event.
This is known as a dual-write consistency problem.
Transactional Outbox Pattern
One common solution is the Transactional Outbox Pattern.
Instead of immediately updating the business database and Kafka independently:
Application | +--> Orders Table | +--> Kafka
write the business record and an outbox entry in the same database transaction.
Database Transaction | +--> Orders | +--> Outbox Events
A separate publisher then publishes the outbox event to Kafka.
Outbox | v Kafka
Conceptually:
Order API | v Database Transaction | +--> ORDER | +--> OUTBOX | v Publisher | v Kafka
This pattern is widely used where losing an event after committing business data would be unacceptable.
Event-Driven Saga Architecture
Kafka can also participate in distributed business workflows.
Consider:
Create Order | v Reserve Inventory | v Process Payment | v Arrange Shipment
Instead of one distributed database transaction, different services perform local transactions and communicate through events.
Example:
OrderCreated | v InventoryReserved | v PaymentCompleted | v ShipmentRequested
If payment fails:
PaymentFailed | v ReleaseInventory
This style is commonly associated with the Saga pattern.
Choreography vs Orchestration
There are two common ways to coordinate distributed workflows.
Choreography
Services react directly to events.
OrderCreated | v Inventory Service | InventoryReserved | v Payment Service
There is no central workflow controller.
Orchestration
A workflow or process engine coordinates activities.
Process Engine | +--> Order Service | +--> Inventory Service | +--> Payment Service
Kafka can still be used for asynchronous events while orchestration controls the overall business process.
For complex enterprise workflows, a hybrid approach is often practical:
BPM / Workflow Engine + Kafka + Microservices
Kafka and BPM Platforms
Kafka is also useful alongside workflow and BPM technologies such as:
Camunda jBPM Flowable
Example:
Customer Application | v Kafka Event | v Camunda Process | +--> Eligibility | +--> Verification | +--> Approval
Or the workflow itself may publish events:
Loan Approved | v Kafka | +---+---+ | | v v CRM Notification
This combination can provide orchestration for long-running workflows while Kafka handles asynchronous system integration.
Kafka Consumer Scaling
Suppose:
Topic = order-events Partitions = 6
You initially run:
2 consumers
Kafka can distribute the partitions between those consumers.
Later traffic increases.
You scale to:
6 consumers
The partitions can then be distributed across more consumer instances.
This is one reason partition planning matters when designing Kafka systems.
What Is Kafka Consumer Lag?
A Kafka consumer processes events using offsets.
Conceptually:
Latest Kafka Offset: 12000 Consumer Offset: 11750 Lag: 250
Consumer lag represents how far a consumer is behind the latest available records.
Growing lag may indicate:
Consumer processing is slow Insufficient consumer capacity Downstream API latency Database bottlenecks Large message processing Application errors
Monitoring consumer lag is therefore important in production Kafka environments.
Monitoring Kafka Microservices
Important metrics include:
Producer throughput Consumer throughput Consumer lag Processing latency Failed messages Retry count Dead-letter events Broker availability Partition health
These can be integrated with observability platforms such as:
Prometheus Grafana OpenTelemetry Enterprise APM tools
Common Kafka Microservices Mistakes
Some frequent design problems include:
Using Kafka for every interaction
Not every request needs asynchronous messaging.
Poor partition-key design
The key affects partitioning and ordering.
Publishing database entities directly
It creates tight schema coupling.
Ignoring duplicate processing
Consumers should be designed with idempotency requirements in mind.
No retry strategy
Failures need controlled retries and recovery handling.
No dead-letter strategy
Repeatedly failing events should be made visible for diagnosis and controlled reprocessing.
Too many unrelated events in one topic
Topic design should reflect business domains and event usage.
Treating Kafka as only a queue
Kafka is fundamentally an event-streaming platform with durable topics, partitions, producers and consumers—not simply a traditional message queue.
Real-World Enterprise Example
Imagine an online loan application platform.
A customer submits an application.
Loan Application Service | | ApplicationSubmitted v Kafka
Several services may consume the event:
Risk Service Credit Service Document Service Notification Service Analytics Service
The Risk Service might later publish:
RiskAssessmentCompleted
The Credit Service might publish:
CreditCheckCompleted
A workflow engine could consume those results and continue the application process.
ApplicationSubmitted | v Risk Assessment | v Credit Check | v Eligibility Decision | v ApplicationApproved
This architecture combines:
Spring Boot Microservices Apache Kafka Business Events Workflow Automation Independent Services
and is a common pattern for building scalable enterprise integration platforms.
Benefits of Kafka + Spring Boot
Using Kafka with Spring Boot provides several practical benefits.
Loose coupling
Producers do not need direct knowledge of every consumer.
Scalability
Topics can be partitioned and consumers can process events in parallel.
Asynchronous processing
Long-running downstream processing does not necessarily need to block the originating request.
Event replay
Retained events can be consumed again when the architecture and retention configuration allow it.
Multiple subscribers
Different consumer groups can independently process the same topic.
Spring integration
Spring Kafka provides abstractions including KafkaTemplate, listener containers and @KafkaListener.
When Should You Use Kafka?
Kafka is a strong option for:
Event-driven microservices Real-time data pipelines Order processing Financial events Workflow integration Audit/event streams Notifications Analytics pipelines IoT events Change Data Capture Asynchronous enterprise integration
When Kafka May Be Unnecessary
Kafka introduces operational and architectural complexity.
For a small application where:
Service A calls Service B
and an immediate response is required, REST may be simpler.
The architecture should solve the business requirement rather than use Kafka merely because it is popular.
Frequently Asked Questions
What is event-driven microservices architecture?
Event-driven microservices communicate by publishing and consuming events instead of relying exclusively on direct synchronous calls between services.
How does Kafka help microservices?
Kafka provides topics through which producers publish events and consumers independently subscribe to those events, reducing direct coupling between applications.
What does Spring Boot provide for Kafka?
Spring Boot provides Kafka auto-configuration through Spring Kafka, including configuration through spring.kafka.*, auto-configured KafkaTemplate, and support for @KafkaListener.
What is KafkaTemplate?
KafkaTemplate is Spring Kafka's high-level abstraction commonly used for sending Kafka records.
What is @KafkaListener?
@KafkaListener creates a Kafka listener endpoint that can receive and process Kafka messages in a Spring application.
Can multiple microservices consume the same Kafka event?
Yes. Different Kafka consumer groups can independently subscribe to the same topic.
Does Kafka guarantee event ordering?
Kafka guarantees ordering within a particular topic-partition.
Should Kafka replace REST APIs?
No. Many enterprise architectures use REST for synchronous request-response interactions and Kafka for asynchronous events.
Conclusion
Event-Driven Microservices with Kafka and Spring Boot provide a powerful architecture for building loosely coupled, scalable and asynchronous enterprise applications.
The fundamental flow is simple:
Producer | v Kafka Topic | v Consumers
But production architectures introduce additional considerations:
Partitions Consumer Groups Offsets Message Keys Idempotency Retries Dead Letter Topics Schema Evolution Transactional Outbox Monitoring
Understanding these concepts is what turns a basic Kafka demo into a reliable enterprise architecture.
Spring Boot simplifies the application-development side with KafkaTemplate, @KafkaListener, external Kafka configuration and Spring Kafka integration, while Apache Kafka provides the distributed event-streaming infrastructure underneath.
Recommended Articles
If you found this guide helpful, share it with your team and follow my blog for more Enterprise Java, Alfresco, Camunda, Flowable, Kafka, Spring Boot, and System Design tutorials.
Learn IT with Shikha covers Apache Kafka, Java, Spring Boot, Microservices, Camunda, BPMN, Alfresco and enterprise software architecture.
📢 Need help with Java, workflows, or backend systems?
I help teams design scalable, high-performance, production-ready applications and solve critical real-world issues.
Services:
- Java & Spring Boot development
- Camunda Training / consulting
- Alfresco Training / consulting
- Workflow architecture guidance
- Workflow implementation (Camunda, Flowable – BPMN, DMN)
- Backend & API integrations (REST, microservices)
- Document management & ECM integrations (Alfresco)
- Performance optimization & production issue resolution
🔗 https://shikhanirankari.blogspot.com/p/professional-services.html
📩 Email: ishikhanirankari@gmail.com | info@realtechnologiesindia.com
🌐 https://realtechnologiesindia.com
✔ Available for quick consultations
✔ Response within 24 hours
🎥 Learn IT with Shikha on YouTube
Prefer learning through videos? Watch practical tutorials on Kafka, Camunda, Alfresco, Java, Spring Boot, Microservices and Enterprise Architecture.▶ Subscribe to Learn IT with Shikha on YouTube
Comments
Post a Comment