Lecture 7: Code an LLM Tokenizer from Scratch in Python

Veröffentlicht am: 01 Januar 1970
auf dem Kanal: Vizuara

112,607

2.3k

In this lecture, we will build a simple tokenizer from scratch in Python.

The key reference book which this video series very closely follows is Build a Large Language Model from Scratch by Manning Publications. All schematics and their descriptions are borrowed from this incredible book!

This book serves as a comprehensive guide to understanding and building large language models, covering key concepts, techniques, and implementations.

Affiliate links for purchasing the book will be added soon. Stay tuned for updates!

0:00 Lecture goals
2:48 2 steps of tokenisation
7:22 Importing the dataset
12:43 Tokenising the text
24:18 Converting tokens into token IDs
31:33 Simple Tokenizer class in Python
43:33 Special Context Tokens
58:33 Additional context tokens
1:02:37 Lecture recap

Entire Code file link: https://drive.google.com/file/d/1_lgi...

=================================================

✉️ Join our FREE Newsletter: https://vizuara.ai/our-newsletter/

=================================================
Vizuara philosophy:

As we learn AI/ML/DL the material, we will share thoughts on what is actually useful in industry and what has become irrelevant. We will also share a lot of information on which subject contains open areas of research. Interested students can also start their research journey there.

Students who are confused or stuck in their ML journey, maybe courses and offline videos are not inspiring enough. What might inspire you is if you see someone else learning and implementing machine learning from scratch.

No cost. No hidden charges. Pure old school teaching and learning.

=================================================

🌟 Meet Our Team: 🌟

🎓 Dr. Raj Dandekar (MIT PhD, IIT Madras department topper)
🔗 LinkedIn:   / raj-abhijit-dandekar-67a33118a

🎓 Dr. Rajat Dandekar (Purdue PhD, IIT Madras department gold medalist)
🔗 LinkedIn:   / rajat-dandekar-901324b1

🎓 Dr. Sreedath Panat (MIT PhD, IIT Madras department gold medalist)
🔗 LinkedIn:   / sreedath-panat-8a03b69a

🎓 Sahil Pocker (Machine Learning Engineer at Vizuara)
🔗 LinkedIn:   / sahil-p-a7a30a8b

🎓 Abhijeet Singh (Software Developer at Vizuara, GSOC 24, SOB 23)
🔗 LinkedIn:   / abhijeet-singh-9a1881192

🎓 Sourav Jana (Software Developer at Vizuara)
🔗 LinkedIn:   / souravjana131

Auf dieser Seite können Sie das Online-Video Lecture 7: Code an LLM Tokenizer from Scratch in Python mit der Dauer stunde minuten sekunde in guter Qualität ansehen, das der Benutzer Vizuara 01 Januar 1970 hochgeladen hat, den Link mit Freunden und Bekannten teilen, dieses Video wurde auf Youtube bereits 112,607 Mal angesehen und es wurde von 2.3 tausend den Zuschauern gefallen. Viel Spaß beim Betrachtenden Zuschauern gefallen!

331