Skip to main content

Apache Kafka : Developing Producer & Consumer in Java

In this article we will see how to implement "Kafka Producer" and "Kafka Consumer" in plain Java project.

1) Download and Install Kafka

Download the latest Kafka release and extract it: 

% wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz
% cd kafka_2.12-3.2.0

Run the following command in order to start ZooKeeper:

% bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal session and run following command in order to start Kafka:

% bin/kafka-server-start.sh config/server.properties

2) Producer

2.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-producer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

2.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

2.3) Kafka Producer

Here is the code for sample "Kafka Producer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers", "client.id", "key.serializer" and "value.serializer".

2.4) Main

This is plain old java driver class; all the execution starts from here.


3) Consumer

3.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-consumer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

3.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

3.3) Kafka Consumer

Here is the code for sample "Kafka Consumer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers, "group.id", "enable.auto.commit", "auto.commit.interval.ms", "session.timeout.ms", "key.deserializer" and "value.serializer".

3.4) Main

This is plain old java driver class; all the execution starts from here.

4) Testing

Here, we are all done with "Producer" and "Consumer" project setup, lets start both applications one bye one.


4.1) Producer Logs

Message with id: '11' sent to partition(0), offset(95) in 59 ms
Message with id: '39' sent to partition(0), offset(96) in 5 ms
Message with id: '1' sent to partition(0), offset(97) in 6 ms
Message with id: '35' sent to partition(0), offset(98) in 5 ms
Message with id: '36' sent to partition(0), offset(99) in 6 ms
Message with id: '41' sent to partition(0), offset(100) in 5 ms
Message with id: '8' sent to partition(0), offset(101) in 6 ms
Message with id: '28' sent to partition(0), offset(102) in 4 ms
Message with id: '29' sent to partition(0), offset(103) in 4 ms
Message with id: '19' sent to partition(0), offset(104) in 6 ms
Message with id: '28' sent to partition(0), offset(105) in 5 ms
Message with id: '41' sent to partition(0), offset(106) in 6 ms

4.2) Consumer Logs

Received message for key: 11, at offset 95
Received message for key: 39, at offset 96
Received message for key: 1, at offset 97
Received message for key: 35, at offset 98
Received message for key: 36, at offset 99
Received message for key: 41, at offset 100
Received message for key: 8, at offset 101
Received message for key: 28, at offset 102
Received message for key: 29, at offset 103
Received message for key: 19, at offset 104
Received message for key: 28, at offset 105
Received message for key: 41, at offset 106

Source Code: Producer, Consumer

Labels:

Comments

Post a Comment

Popular posts from this blog

Deploying Spring Boot microservices on Kubernetes Cluster

This article guides you through the deployment of two Spring Boot microservices, namely "order-service" and "inventory-service," on Kubernetes using "MiniKube" . We will establish communication between them, with "order-service" making calls to an endpoint in "inventory-service." Additionally, we will configure "order-service" to be accessible from the local machine's browser . 1) Create Spring Boot microservices The Spring Boot microservices, "order-service" and "inventory-service," have been developed and can be found in this GitHub repository. If you are interested in learning more about creating Spring Boot REST microservices, please refer to this or this (Reactive) link. 2) Build Docker Images The Docker images for both "order-service" and "inventory-service" have already been generated and deployed on DockerHub, as shown below. codeburps/order-service cod...
Post a Comment
Read more

Circuit Breaker Pattern with Resilience4J in a Spring Boot Application

Read Also: Spring Cloud Circuit Breaker + Resilience4j Resilience4j is a lightweight fault tolerance library that draws inspiration from Netflix Hystrix but is specifically crafted for functional programming. The library offers higher-order functions, known as decorators , designed to augment any functional interface, lambda expression, or method reference with features such as Circuit Breaker, Rate Limiter, Retry, or Bulkhead . These functionalities can be seamlessly integrated within a project, class, or even applied to a single method. It's possible to layer multiple decorators on any functional interface, lambda expression, or method reference, allowing for versatile and customizable fault tolerance. While numerous annotation-based implementations exist online, this article focuses solely on the reactive approach using router predicates and router functions . How Circuit Breaker Pattern works? In general, a circuit breaker functions as an automatic electrical s...
Post a Comment
Read more

How to create a basic Spring 6 project using Maven

Below is a step-by-step guide to creating a basic Spring project using Maven. 1) Create a Maven Project Use the following Maven command to create a new Maven project. mvn archetype:generate -DgroupId=com.tb -DartifactId=spring-demo -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false 2) Import in IntelliJ IDEA If you haven't already, open IntelliJ IDEA on your system. Go to "File" > "New" > "Project from Existing Sources..." . In the file dialog, navigate to the directory where your Maven project is located. Select the pom.xml file within the project directory and click "Open." 3) Update pom.xml In total, the application requires the below-mentioned dependencies: 4) Create Spring Configuration Create a Java configuration class that uses annotations to define your Spring beans and their dependencies. This class should be annotated with @Configuration . 5) Create the Main Application C...
Post a Comment
Read more