Processing Complex Type & Producing Messages to Topic using Faust
This video covers how to consume complex Types from Input Kafka Topic & Publish the messages in an Output Topic after applying the required transformation using Faust.
Prerequisite:
-------------------------
Introduction to Kafka Stream Processing in Python using Faust
• Introduction to Kafka Stream Processi...
Documentation Link:
------------------------------------
https://faust.readthedocs.io/en/latest/
Start Zookeeper:
------------------------------
F:/kafka_2.12-3.3.1/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.3.1/config/zookeeper.properties
Start Broker:
------------------------------
F:/kafka_2.12-3.3.1/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.3.1/config/server.properties
Start Kafka Topic:
------------------------------
F:/kafka_2.12-3.3.1/bin/windows/kafka-topics.bat --create --topic hello_world --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
F:/kafka_2.12-3.3.1/bin/windows/kafka-topics.bat --create --topic send_greetings --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
Produce Messages:
------------------------------
F:/kafka_2.12-3.3.1/bin/windows/kafka-console-producer.bat --topic hello_world --bootstrap-server localhost:9092
faust -A main worker -l info
Start Kakfa Consumer:
--------------------------
F:/kafka_2.12-3.3.1/bin/windows/kafka-console-consumer.bat --topic send_greetings --bootstrap-server localhost:9092
Code:
--------------------
import faust
app=faust.App('demo-streaming',broker='localhost:9092')
class Greeting(faust.Record,serializer='json'):
from_name:str
to_name:str
input_topic = app.topic('hello_world', value_type=Greeting)
output_topic = app.topic('send_greetings', value_type=str,value_serializer='raw')
@app.agent(input_topic)
async def processor(stream):
async for message in stream:
output_tf=f'Greeting from {message.from_name} to {message.to_name}'
print(output_tf)
await output_topic.send(value=output_tf)
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engineering with Cl...
Apache Kafka form scratch
• Apache Kafka for Python Developers
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL
In questa pagina del sito puoi guardare il video online Processing Complex Type & Producing Messages to Topic using Faust della durata di ore minuti seconda in buona qualità , che l'utente ha caricato Knowledge Amplifier 18 febbraio 2023, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 1,103 volte e gli è piaciuto 25 spettatori. Buona visione!
