Skip to main content

Apache Kafka : Developing Producer & Consumer in Java

In this article we will see how to implement "Kafka Producer" and "Kafka Consumer" in plain Java project.

1) Download and Install Kafka

Download the latest Kafka release and extract it: 

% wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz
% cd kafka_2.12-3.2.0

Run the following command in order to start ZooKeeper:

% bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal session and run following command in order to start Kafka:

% bin/kafka-server-start.sh config/server.properties

2) Producer

2.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-producer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

2.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

2.3) Kafka Producer

Here is the code for sample "Kafka Producer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers", "client.id", "key.serializer" and "value.serializer".

2.4) Main

This is plain old java driver class; all the execution starts from here.


3) Consumer

3.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-consumer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

3.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

3.3) Kafka Consumer

Here is the code for sample "Kafka Consumer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers, "group.id", "enable.auto.commit", "auto.commit.interval.ms", "session.timeout.ms", "key.deserializer" and "value.serializer".

3.4) Main

This is plain old java driver class; all the execution starts from here.

4) Testing

Here, we are all done with "Producer" and "Consumer" project setup, lets start both applications one bye one.


4.1) Producer Logs

Message with id: '11' sent to partition(0), offset(95) in 59 ms
Message with id: '39' sent to partition(0), offset(96) in 5 ms
Message with id: '1' sent to partition(0), offset(97) in 6 ms
Message with id: '35' sent to partition(0), offset(98) in 5 ms
Message with id: '36' sent to partition(0), offset(99) in 6 ms
Message with id: '41' sent to partition(0), offset(100) in 5 ms
Message with id: '8' sent to partition(0), offset(101) in 6 ms
Message with id: '28' sent to partition(0), offset(102) in 4 ms
Message with id: '29' sent to partition(0), offset(103) in 4 ms
Message with id: '19' sent to partition(0), offset(104) in 6 ms
Message with id: '28' sent to partition(0), offset(105) in 5 ms
Message with id: '41' sent to partition(0), offset(106) in 6 ms

4.2) Consumer Logs

Received message for key: 11, at offset 95
Received message for key: 39, at offset 96
Received message for key: 1, at offset 97
Received message for key: 35, at offset 98
Received message for key: 36, at offset 99
Received message for key: 41, at offset 100
Received message for key: 8, at offset 101
Received message for key: 28, at offset 102
Received message for key: 29, at offset 103
Received message for key: 19, at offset 104
Received message for key: 28, at offset 105
Received message for key: 41, at offset 106

Source Code: Producer, Consumer

Labels:

Comments

Post a Comment

Popular posts from this blog

Deploying Spring Boot microservices on Kubernetes Cluster

This article guides you through the deployment of two Spring Boot microservices, namely "order-service" and "inventory-service," on Kubernetes using "MiniKube" . We will establish communication between them, with "order-service" making calls to an endpoint in "inventory-service." Additionally, we will configure "order-service" to be accessible from the local machine's browser . 1) Create Spring Boot microservices The Spring Boot microservices, "order-service" and "inventory-service," have been developed and can be found in this GitHub repository. If you are interested in learning more about creating Spring Boot REST microservices, please refer to this or this (Reactive) link. 2) Build Docker Images The Docker images for both "order-service" and "inventory-service" have already been generated and deployed on DockerHub, as shown below. codeburps/order-service cod...
Post a Comment
Read more

Circuit Breaker Pattern with Resilience4J in a Spring Boot Application

Read Also: Spring Cloud Circuit Breaker + Resilience4j Resilience4j is a lightweight fault tolerance library that draws inspiration from Netflix Hystrix but is specifically crafted for functional programming. The library offers higher-order functions, known as decorators , designed to augment any functional interface, lambda expression, or method reference with features such as Circuit Breaker, Rate Limiter, Retry, or Bulkhead . These functionalities can be seamlessly integrated within a project, class, or even applied to a single method. It's possible to layer multiple decorators on any functional interface, lambda expression, or method reference, allowing for versatile and customizable fault tolerance. While numerous annotation-based implementations exist online, this article focuses solely on the reactive approach using router predicates and router functions . How Circuit Breaker Pattern works? In general, a circuit breaker functions as an automatic electrical s...
Post a Comment
Read more

Reactive programming in Java with Project Reactor

Reactive programming is a declarative programming paradigm that focuses on building applications that are responsive, resilient, and scalable in the face of modern challenges like concurrency, distributed systems, and asynchronous data streams . Reactive programming provides a set of tools, patterns, and abstractions to handle asynchronous and event-driven programming more effectively. Imperative programming focuses on describing the step-by-step instructions or commands that the computer needs to follow to achieve a specific task. In this paradigm, you explicitly state how to perform each operation and control flow in your code. The emphasis is on "how" the computation should be done. int sum = 0; for (int i = 1; i Declarative programming emphasizes specifying what you want to achieve rather than detailing how to achieve it. You describe the desired outcome or the properties of the result, and the programming language or framework handles the execution detai...
Post a Comment
Read more