Kafka Consumer using Python & Concept of Offset-Commit

Published: 11 September 2022
on channel: Knowledge Amplifier

3,968

In this video, Kafka Consumer using Python & the concept of Offset-Commit is explained in detail.

Prerequisite:
-------------------------
Consumer & Consumer Group in Kafka
   • Consumer & Consumer Group in Kafka
Kafka Consumer Groups CLI Demo | Kafka-Python
   • Kafka Consumer Groups CLI Demo | Kafk...

Producer Code:
------------------------------------
from time import sleep
from json import dumps
from kafka import KafkaProducer

topic_name='hello_world4'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'))

for e in range(1000):
data = {'number' : e}
print(data)
producer.send(topic_name, value=data)
sleep(2)

Consumer Code:
---------------------------------------
from kafka import KafkaConsumer

import json

consumer = KafkaConsumer ('hello_world4',bootstrap_servers = ['localhost:9092'],
value_deserializer=lambda m: json.loads(m.decode('utf-8')),group_id='demo112215sgtrjwrykvjh',auto_offset_reset='earliest')

for message in consumer:
print(message)

Check this playlist for more Data Engineering related videos:
   • Demystifying Data Engineering with Cl...

Apache Kafka form scratch
   • Apache Kafka for Python Developers

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

On this page of the site you can watch the video online Kafka Consumer using Python & Concept of Offset-Commit with a duration of hours minute second in good quality, which was uploaded by the user Knowledge Amplifier 11 September 2022, share the link with friends and acquaintances, this video has already been watched 3,968 times on youtube and it was liked by 73 viewers. Enjoy your viewing!

1,572