Modern enterprise systems rarely keep all their data in one application.

Customer information may reside in PostgreSQL, orders in MySQL, events in Apache Kafka, analytics data in a warehouse, and searchable documents in Elasticsearch.

Moving data reliably between all these systems can quickly become complicated.

This is exactly the problem Apache Kafka Connect is designed to solve.

Kafka Connect provides a standardized framework for moving data into and out of Apache Kafka without requiring every development team to build and maintain custom producers and consumers for common integration scenarios. Kafka Connect supports both standalone and distributed deployment models and provides automatic offset management and a REST interface for connector management.

In this guide, we'll explore:

Kafka Connect architecture
Source Connectors
Sink Connectors
Connect Workers
Connectors and Tasks
Converters
Single Message Transforms
Offset management
Standalone vs distributed mode
Change Data Capture (CDC)
Debezium with Kafka Connect
Error handling
Scaling and fault tolerance
Production architecture and best practices

French version: https://shikhanirankari.blogspot.com/2026/09/architecture-kafka-connect-source-sink-cdc.html

1. What Is Kafka Connect?

Kafka Connect is an integration framework for moving data between Apache Kafka and external systems.

Instead of writing custom integration code for every database, file system, search platform or data store, you configure an appropriate connector.

The basic architecture is:


External Source
      ↓
Source Connector
      ↓
Kafka Connect
      ↓
Kafka Topics
      ↓
Kafka Connect
      ↓
Sink Connector
      ↓
External Target

Apache Kafka defines two fundamental connector types:

Source Connector

Moves data from an external system into Kafka.

Sink Connector

Moves data from Kafka to an external system.

2. Why Do We Need Kafka Connect?

Imagine that an organization needs to integrate:


PostgreSQL → Kafka
MySQL → Kafka
Kafka → Elasticsearch
Kafka → Data Warehouse
Kafka → Object Storage

One approach is to develop a custom application for every integration.

That means managing:

Kafka producers
Kafka consumers
serialization
offsets
retries
failures
scaling
monitoring
deployment
configuration

Kafka Connect provides a common framework for handling much of this integration infrastructure.

This lets developers focus on what data should move and where, rather than repeatedly implementing the plumbing.

3. Kafka Connect Architecture

A Kafka Connect environment contains several important components:


External System
      ↓
Connector
      ↓
Tasks
      ↓
Worker
      ↓
Converters / SMTs
      ↓
Kafka Cluster

The most important concepts are:

Component	Responsibility
Connector	Defines the integration
Task	Performs data movement
Worker	Executes connectors and tasks
Converter	Converts data representation
SMT	Performs lightweight record transformation
Kafka Topics	Store the streamed records
Offsets	Track processing progress
REST API	Manages connectors

A connector does not itself perform all the data copying. Kafka's connector model divides the work into Tasks, which can be distributed among workers.

4. What Is a Kafka Connect Worker?

A worker is a running Kafka Connect process.

Workers execute:

connectors
tasks
converters
transformations

Think of the relationship as:


Kafka Connect Cluster
        |
        +---- Worker 1
        |       |
        |       +---- Task A
        |       +---- Task B
        |
        +---- Worker 2
        |       |
        |       +---- Task C
        |
        +---- Worker 3
                |
                +---- Task D

When Kafka Connect runs in distributed mode, multiple workers can cooperate as a Connect cluster.

Adding workers can increase capacity and improve availability because work can be redistributed across available workers. Kafka's documentation describes Kafka Connect as distributed and scalable, using Kafka's group-management mechanisms.

5. Connectors vs Tasks

This distinction is frequently misunderstood.

Connector

The Connector manages the integration configuration and determines how work should be divided.

Task

A Task performs the actual data movement.

For example:


JDBC Source Connector
        |
        +---- Task 1
        +---- Task 2
        +---- Task 3

The configuration:


{
  "tasks.max": "3"
}

means the connector may create up to three tasks.

It does not necessarily guarantee that three tasks will be created; the connector may create fewer if the underlying source cannot be parallelized to that degree.

Kafka Connect workers connectors and tasks architecture in distributed mode

6. What Is a Source Connector?

A Source Connector imports data from an external system into Kafka.

The direction is:


External System
       ↓
Source Connector
       ↓
Kafka Topic

The Kafka API describes SourceConnector as the connector abstraction for pulling data from another system and sending it to Kafka.

Common source systems include:

relational databases
files
SaaS applications
message systems
cloud storage
application logs
CDC platforms

For example:


PostgreSQL
     ↓
Debezium PostgreSQL Connector
     ↓
Kafka Connect
     ↓
Kafka Topic

7. Source Connector Example

A conceptual source connector configuration could look like:


{
  "name": "customer-source",
  "config": {
    "connector.class": "com.example.CustomerSourceConnector",
    "tasks.max": "2",
    "topic": "customers"
  }
}

Important common configuration properties include:


name
connector.class
tasks.max
key.converter
value.converter

Connector-specific properties depend on the connector being used.

8. What Is a Sink Connector?

A Sink Connector moves records from Kafka topics to an external destination.

The direction is:


Kafka Topic
      ↓
Sink Connector
      ↓
Target System

Kafka's Sink API defines sink connectors as connectors that send Kafka data to another system.

Typical targets include:

databases
Elasticsearch
data warehouses
object storage
analytics systems
file systems
external services

For example:


Kafka Topic
     ↓
Sink Connector
     ↓
Analytics Database

9. Sink Connector Example

A simplified configuration could look like:


{
  "name": "customer-sink",
  "config": {
    "connector.class": "com.example.CustomerSinkConnector",
    "tasks.max": "2",
    "topics": "customers"
  }
}

Sink connectors typically subscribe to one or more Kafka topics.

Kafka Connect also supports using a topic regular expression in appropriate configurations, allowing a sink to consume records from topics matching a defined pattern.

10. Source vs Sink Connectors

The easiest way to remember the difference is to think from Kafka's perspective.

Source Connector	Sink Connector
External system → Kafka	Kafka → External system
Produces records to Kafka	Consumes records from Kafka
Reads source data	Writes destination data
Example: Database → Kafka	Example: Kafka → Elasticsearch
Can be used for CDC	Commonly used for downstream delivery

So:

SOURCE = INTO Kafka

SINK = OUT OF Kafka

11. End-to-End Kafka Connect Pipeline

A complete integration might look like:


PostgreSQL
     ↓
Debezium Source Connector
     ↓
Kafka Connect
     ↓
customer-events
     ↓
Kafka Cluster
     ↓
Sink Connector
     ↓
Analytics / Search / Data Warehouse

This architecture allows the source and destination systems to remain decoupled.

The source does not need to know which downstream systems consume its changes.

Likewise, multiple applications can consume the same Kafka events independently.

12. What Are Kafka Connect Converters?

Connectors move records, but the representation of those records must also be handled.

This is where Converters become important.

Typical data formats include:


JSON
Avro
Protobuf
String
ByteArray

Worker or connector configuration may define:


key.converter=...
value.converter=...

The converter determines how Kafka Connect data is represented when written to or read from Kafka.

This separation is useful because connector logic does not have to be tightly coupled to one serialization format.

13. What Are Single Message Transforms (SMTs)?

Kafka Connect provides Single Message Transforms, commonly called SMTs.

SMTs perform lightweight transformations on individual records while they move through Kafka Connect.

Conceptually:


Source
   ↓
Connector
   ↓
SMT
   ↓
Converter
   ↓
Kafka

or on the sink side:


Kafka
   ↓
Converter
   ↓
SMT
   ↓
Sink Connector
   ↓
Destination

Kafka Connect allows a transformation chain to be configured and applies the transformations in the configured order.

Typical use cases include:

renaming fields
removing fields
adding metadata
routing records
modifying topic names
extracting parts of a record

SMTs are intended for relatively lightweight, record-level operations.

Complex stream processing is usually better handled using technologies such as Kafka Streams or another stream-processing platform.

14. What Is Change Data Capture (CDC)?

Change Data Capture, or CDC, is a technique for detecting changes occurring in a database and propagating those changes to other systems.

Suppose a customer record changes from:


status = PENDING

to:


status = APPROVED

Instead of repeatedly querying the entire customer table, a CDC system can capture the database change and publish an event.

Conceptually:


Database Transaction
        ↓
Transaction / Replication Log
        ↓
CDC Connector
        ↓
Kafka Connect
        ↓
Kafka Topic

CDC is especially useful for:

event-driven architectures
microservices
analytics
cache synchronization
search indexing
data replication
audit pipelines
data warehouses

15. Debezium + Kafka Connect

Debezium is widely used to implement CDC with Kafka Connect.

Debezium connectors monitor database changes and publish change events.

A typical architecture is:


MySQL / PostgreSQL
        ↓
Database Change Log
        ↓
Debezium Source Connector
        ↓
Kafka Connect
        ↓
Kafka Topics

Debezium's architecture documentation explains that Debezium is commonly deployed through Kafka Connect as source connectors, while sink connectors can then propagate those events from Kafka into downstream systems.

For example, the MySQL connector can access the database's binlog, while PostgreSQL CDC can use a logical replication stream.

Kafka Connect Debezium CDC architecture for MySQL and PostgreSQL change data capture

16. Why Log-Based CDC?

A traditional integration may periodically execute:


SELECT *
FROM customers
WHERE updated_at > ?

This is a polling approach.

CDC takes a different approach by reading database change information.

Debezium's documentation highlights several advantages of its log-based CDC approach, including capturing deletes, avoiding the CPU overhead of frequent polling, supporting low-delay change propagation, and avoiding the need for an updated_at-style column solely for polling.

This makes CDC particularly attractive for near-real-time integration.

17. Example CDC Event

Suppose this record changes:


Customer ID: 101
Name: John
Status: PENDING

to:


Customer ID: 101
Name: John
Status: APPROVED

A CDC event may conceptually contain:


{
  "before": {
    "id": 101,
    "name": "John",
    "status": "PENDING"
  },
  "after": {
    "id": 101,
    "name": "John",
    "status": "APPROVED"
  },
  "op": "u"
}

The precise event structure depends on the connector and configuration.

Downstream consumers can now react to the change without querying the operational database.

18. Initial Snapshot + Continuous CDC

A common CDC requirement is:

What happens to data that already existed before the connector started?

Many CDC implementations support an initial snapshot followed by continuous change capture.

Conceptually:


STEP 1
Existing Database
      ↓
Initial Snapshot
      ↓
Kafka

STEP 2
New INSERT / UPDATE / DELETE
      ↓
Database Log
      ↓
CDC
      ↓
Kafka

Debezium supports snapshot capabilities so an initial database state can be captured when appropriate before continuing with ongoing changes.

19. Kafka Connect Offset Management

Kafka Connect needs to know how much data has already been processed.

This is handled through offsets.

For a source connector, an offset might represent:

file position
database log position
sequence number
timestamp
source-specific position

Conceptually:


Record 1001  ✓
Record 1002  ✓
Record 1003  ✓
Record 1004  ← current position
Record 1005

Offset tracking helps connectors resume processing after restarts instead of blindly starting from the beginning.

Automatic offset management is one of the capabilities provided by the Kafka Connect framework.

20. Standalone vs Distributed Kafka Connect

Kafka Connect supports two major deployment modes.

Standalone Mode


Single Connect Process
       ↓
Connectors
       ↓
Tasks

Advantages:

simple setup
useful for development
easy local testing

Limitations:

single process
limited resilience compared with a distributed cluster

Distributed Mode


Kafka Connect Cluster
     |
     +-- Worker 1
     +-- Worker 2
     +-- Worker 3

Advantages:

scalability
workload distribution
improved fault tolerance
centralized connector management
better fit for enterprise production environments

Kafka's documentation describes standalone as a single-process model, while distributed mode supports a cluster of workers.

21. Kafka Connect Internal Topics

In distributed deployments, Kafka Connect maintains internal state in Kafka topics.

Common internal topics conceptually include:


connect-configs
connect-offsets
connect-status

They are used for:

Configurations

Connector configuration information.

Offsets

Progress information.

Status

Connector and task status.

These internal topics are critical operational components of a distributed Kafka Connect deployment.

22. Kafka Connect REST API

Kafka Connect exposes a REST API for managing connectors.

Typical operations include:


GET /connectors

List connectors.


POST /connectors

Create a connector.


GET /connectors/{name}/status

Check connector status.


PUT /connectors/{name}/config

Update configuration.


DELETE /connectors/{name}

Delete a connector.

This makes it possible to integrate connector management into deployment automation and operational tooling.

23. Error Handling and Dead Letter Queues

Real production data is rarely perfect.

A record might contain:

invalid JSON
unexpected schema
incorrect data type
missing required field
serialization problem

Depending on connector and Kafka Connect configuration, error-handling strategies can include:


Retry
Skip
Log
Dead Letter Queue
Fail Connector

A common enterprise design is:


Kafka Topic
     ↓
Sink Connector
     ↓
Bad Record?
  ↙       ↘
No        Yes
↓          ↓
Target     DLQ Topic

A Dead Letter Queue (DLQ) allows problematic records to be isolated for investigation instead of necessarily blocking the entire data pipeline.

24. Kafka Connect Production Architecture

A more realistic enterprise deployment could look like:


                  DATABASES
          MySQL / PostgreSQL / Oracle
                     ↓
              CDC Connectors
                     ↓
          ┌─────────────────────┐
          │ Kafka Connect       │
          │ Worker 1            │
          │ Worker 2            │
          │ Worker 3            │
          └─────────────────────┘
                     ↓
              Kafka Cluster
          Broker 1 / 2 / 3
                     ↓
       ┌─────────────┼─────────────┐
       ↓             ↓             ↓
   Search Sink   Warehouse Sink   Applications

Production environments should additionally consider:

authentication
TLS
ACLs
secret management
monitoring
retries
DLQs
connector versioning
schema compatibility
capacity planning
disaster

Enterprise Kafka Connect production architecture with source sink connectors workers CDC and monitoring

25. Kafka Connect vs Kafka Producers and Consumers

Kafka Connect does not replace all Kafka producers and consumers.

Use Kafka Connect when your requirement primarily involves standardized integration with an external system.

For example:


Database → Kafka
Kafka → Search
Kafka → Data Warehouse

Use a custom producer or consumer when your application contains significant domain-specific processing logic.

For example:


Order Service
     ↓
Complex Business Rules
     ↓
Kafka Producer

A good architecture often uses both.

26. Kafka Connect vs Kafka Streams

These technologies solve different problems.

Kafka Connect	Kafka Streams
Data integration	Stream processing
Moves data	Processes data
Source/Sink connectors	Java stream-processing API
External systems ↔ Kafka	Kafka topics ↔ processing ↔ Kafka topics
Configuration-driven	Application-code-driven

For example:


PostgreSQL
     ↓
Kafka Connect
     ↓
orders
     ↓
Kafka Streams
     ↓
validated-orders
     ↓
Kafka Connect
     ↓
Analytics Platform

This combination is common in event-driven architectures.

27. Kafka Connect + CDC + Microservices

CDC becomes especially useful when modernizing legacy applications.

Imagine an existing application that writes orders directly to PostgreSQL.

Instead of immediately modifying that application to publish Kafka events:


Legacy Application
       ↓
PostgreSQL
       ↓
Database Log
       ↓
Debezium
       ↓
Kafka Connect
       ↓
order-events
       ↓
Microservices

This can allow new event-driven services to react to database changes while reducing changes required in the legacy application.

28. Monitoring Kafka Connect

A production Kafka Connect environment should be monitored continuously.

Important areas include:

worker availability
connector state
task state
failed tasks
source lag
sink lag
throughput
retry rates
DLQ volume
JVM memory
CPU
network
connector errors

A connector showing:


RUNNING

doesn't automatically prove that the entire pipeline is healthy.

Always monitor end-to-end data flow.

29. Kafka Connect Best Practices

For production deployments:

Use distributed mode where resilience and scaling are required.

Run multiple workers to avoid relying on one process.

Monitor connectors and tasks independently.

Protect Kafka Connect's REST API.

Use appropriate converters and schema strategy.

Configure retries and error handling deliberately.

Monitor DLQ topics.

Version connector configurations.

Secure credentials instead of hard-coding passwords.

Test connector upgrades before Production.

Capacity-test CDC against realistic database workloads.

Document source-to-topic and topic-to-target mappings.

30. Common Kafka Connect Interview Questions

What is Kafka Connect?

Kafka Connect is a framework for streaming data between Apache Kafka and external systems.

What is a Source Connector?

A Source Connector moves data from an external system into Kafka.

What is a Sink Connector?

A Sink Connector moves data from Kafka to an external system.

What is a Kafka Connect Worker?

A worker is a Kafka Connect process that executes connectors and tasks.

What is a Task?

A Task performs the actual data-transfer work assigned by a connector.

What is CDC?

Change Data Capture identifies database changes and propagates them to downstream systems.

What is Debezium?

Debezium provides connectors commonly used to capture database changes and publish them as events, frequently through Kafka Connect.

What is an SMT?

A Single Message Transform performs lightweight per-record transformations within Kafka Connect.

Standalone or distributed mode for Production?

For resilient enterprise deployments, distributed mode is generally the appropriate architecture.

31. Kafka Connect Architecture — Quick Summary

Remember the architecture like this:


                SOURCE SIDE

Database / File / API
        ↓
Source Connector
        ↓
Source Tasks
        ↓
Kafka Connect Workers
        ↓
Converters + SMT
        ↓
Kafka Topics


                 SINK SIDE

Kafka Topics
        ↓
Converters + SMT
        ↓
Kafka Connect Workers
        ↓
Sink Tasks
        ↓
Sink Connector
        ↓
Database / Search / Warehouse

And for CDC:


Database
   ↓
Transaction Log
   ↓
Debezium
   ↓
Kafka Connect
   ↓
Kafka Topics
   ↓
Downstream Systems

Conclusion

Kafka Connect provides a standardized and scalable approach for integrating Apache Kafka with databases, file systems, search platforms, analytics systems and other external technologies.

The architecture becomes much easier to understand once you separate its major responsibilities:

Source Connectors bring data into Kafka.

Sink Connectors move data out of Kafka.

Connectors define and coordinate integration work.

Tasks perform the actual data movement.

Workers execute connectors and tasks.

Converters control data representation.

SMTs perform lightweight record transformations.

And when Kafka Connect is combined with Debezium, it becomes a powerful foundation for Change Data Capture (CDC) architectures that stream database changes into Kafka. Debezium's standard Kafka-based architecture uses source connectors to capture database changes, after which sink connectors and other consumers can deliver or process those events downstream.

For enterprise event-driven architectures, this combination provides a clean way to connect operational databases, Kafka, microservices, analytics systems and downstream data platforms.

Kafka Connect Architecture Explained: Source Connectors, Sink Connectors & CDC

1. What Is Kafka Connect?

2. Why Do We Need Kafka Connect?

3. Kafka Connect Architecture

4. What Is a Kafka Connect Worker?

5. Connectors vs Tasks

Connector

Task

6. What Is a Source Connector?

7. Source Connector Example

8. What Is a Sink Connector?

9. Sink Connector Example

10. Source vs Sink Connectors

11. End-to-End Kafka Connect Pipeline

12. What Are Kafka Connect Converters?

13. What Are Single Message Transforms (SMTs)?

14. What Is Change Data Capture (CDC)?

15. Debezium + Kafka Connect

16. Why Log-Based CDC?

17. Example CDC Event

18. Initial Snapshot + Continuous CDC

19. Kafka Connect Offset Management

20. Standalone vs Distributed Kafka Connect

Standalone Mode

Distributed Mode

21. Kafka Connect Internal Topics

22. Kafka Connect REST API

23. Error Handling and Dead Letter Queues

24. Kafka Connect Production Architecture

25. Kafka Connect vs Kafka Producers and Consumers

26. Kafka Connect vs Kafka Streams

27. Kafka Connect + CDC + Microservices

28. Monitoring Kafka Connect

29. Kafka Connect Best Practices

30. Common Kafka Connect Interview Questions

What is Kafka Connect?

What is a Source Connector?

What is a Sink Connector?

What is a Kafka Connect Worker?

What is a Task?

What is CDC?

What is Debezium?

What is an SMT?

Standalone or distributed mode for Production?

31. Kafka Connect Architecture — Quick Summary

Conclusion

Recommended Articles

🎥 Learn IT with Shikha on YouTube

📢 Need help with Java, workflows, or backend systems?

I help teams design scalable, high-performance, production-ready applications and solve critical real-world issues.

Services:

🎥 Learn IT with Shikha on YouTube

Comments

Post a Comment

Popular posts from this blog

Top 50 Camunda BPM Interview Questions and Answers for Developers (2026 Guide)

10 BPMN Best Practices Every Camunda Developer Should Know

OOPs Concepts in Java | English | Object Oriented Programming Explained