Byte Pair Encoding Tokenization in NLP

Published: 22 July 2022
on channel: TechViz - The Data Science Guy

9,767

#tokenization #transformers #nlp
Tokenization is the process of representing text into smaller meaningful lexical units. Byte Pair Encoding (BPE) is a popular subword-based tokenization algorithm used by state-of-the-art NLP models like RoBerta, BART, GPT, etc. In this video, we look into pros and cons of other methods and understand BPE through an example.

⏩ OUTLINE:
0:00 - Tokenization in NLP and it's types.
01:10 - Subword-level Tokenization
01:43 - Byte Pair Encoding (BPE) Algorithm

Enjoy reading articles? then consider subscribing to Medium membership, it is just 5$ a month for unlimited access to all free/paid content.
Subscribe now -   / membership

*********************************************
If you want to support me financially which is totally optional and voluntary :) ❤️
You can consider buying me chai ( because I don't drink coffee :) ) at https://www.buymeacoffee.com/TechvizC...
*********************************************

⏩ IMPORTANT LINKS
Tokenization methods in NLP:    • WordPiece Tokenization in NLP
Research Paper Summaries:    • Simple Unsupervised Keyphrase Extraction u...

*********************************************
⏩ Youtube -    / @techvizthedatascienceguy
⏩ LinkedIn -   / prakhar21
⏩ Medium -   / prakhar.mishra
⏩ GitHub - https://github.com/prakhar21
*********************************************

⏩ Please feel free to share out the content and subscribe to my channel -    / @techvizthedatascienceguy

Tools I use for making videos :)
⏩ iPad - https://tinyurl.com/y39p6pwc
⏩ Apple Pencil - https://tinyurl.com/y5rk8txn
⏩ GoodNotes - https://tinyurl.com/y627cfsa

#techviz #datascienceguy #naturallanguageprocessing #machinelearning #ai

About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3+ years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).

On this page of the site you can watch the video online Byte Pair Encoding Tokenization in NLP with a duration of hours minute second in good quality, which was uploaded by the user TechViz - The Data Science Guy 22 July 2022, share the link with friends and acquaintances, this video has already been watched 9,767 times on youtube and it was liked by 96 viewers. Enjoy your viewing!

87,017

4.1K