LLM “Patchnization”

Published: 02 March 2024
on channel: Jaward Sesay

744

LLM “Patchnization”

Minimal, clean code for video/image "patchnization" - a process commonly applied in tokenizing visual data for use in a Transformer encoder.

Code: https://github.com/Jaykef/min-patchnizer

The code above, first extracts still images (frames) from a video, splits the image frames into smaller fixed-size patches, linearly embeds each of them, adds position embeddings and then saves the resulting sequence of vectors for use in a Vision Transformer encoder. I tried training the resulting sequence vectors with Karpathy's minbpe and it took ~30s per frame to tokenize. The whole "patchnization" took ~10s for a 20s video on my M2 Air.

The files in the repo work as follows:

1. patchnizer.py: Holds code for simple implemenatation of the three stages involved (extract_image_frames from video, reduce image_frames_to_patches of fixed sizes 16x16 pixels, then linearly_embed_patches into a 1D vector sequence with additional position embeddings.

2. patchnize.py: performs the whole process with custom configs (patch_size, created dirs, video - I am using the "dogs playing in snow" video by sora).

3. train.py: Trains the resulting one-dimensional vector sequence (linear_patch_embeddings + position_embeddings) on Karpathy's minbpe (a minimal implementation of the byte-pair encoding algorithm).

4. check.py: Checks to see if the patch embeddings match the original image patches - this basically just do the reverse of linear embedding.

The whole process builds on the approach introduced in the Vision Transformer paper: "An image is worth 16x16 words: Transformers for image recognition at scale."
Paper: https://arxiv.org/abs/2010.11929

#llm #sora #nlp #openai

On this page of the site you can watch the video online LLM “Patchnization” with a duration of hours minute second in good quality, which was uploaded by the user Jaward Sesay 02 March 2024, share the link with friends and acquaintances, this video has already been watched 744 times on youtube and it was liked by 26 viewers. Enjoy your viewing!

3,763