LLM “Patchnization”
LLM “Patchnization”
Minimal, clean code for video/image "patchnization" - a process commonly applied in tokenizing visual data for use in a Transformer encoder.
Code: https://github.com/Jaykef/min-patchnizer
The code above, first extracts still images (frames) from a video, splits the image frames into smaller fixed-size patches, linearly embeds each of them, adds position embeddings and then saves the resulting sequence of vectors for use in a Vision Transformer encoder. I tried training the resulting sequence vectors with Karpathy's minbpe and it took ~30s per frame to tokenize. The whole "patchnization" took ~10s for a 20s video on my M2 Air.
The files in the repo work as follows:
1. patchnizer.py: Holds code for simple implemenatation of the three stages involved (extract_image_frames from video, reduce image_frames_to_patches of fixed sizes 16x16 pixels, then linearly_embed_patches into a 1D vector sequence with additional position embeddings.
2. patchnize.py: performs the whole process with custom configs (patch_size, created dirs, video - I am using the "dogs playing in snow" video by sora).
3. train.py: Trains the resulting one-dimensional vector sequence (linear_patch_embeddings + position_embeddings) on Karpathy's minbpe (a minimal implementation of the byte-pair encoding algorithm).
4. check.py: Checks to see if the patch embeddings match the original image patches - this basically just do the reverse of linear embedding.
The whole process builds on the approach introduced in the Vision Transformer paper: "An image is worth 16x16 words: Transformers for image recognition at scale."
Paper: https://arxiv.org/abs/2010.11929
#llm #sora #nlp #openai
Auf dieser Seite können Sie das Online-Video LLM “Patchnization” mit der Dauer stunde minuten sekunde in guter Qualität ansehen, das der Benutzer Jaward Sesay 02 März 2024 hochgeladen hat, den Link mit Freunden und Bekannten teilen, dieses Video wurde auf Youtube bereits 744 Mal angesehen und es wurde von 26 den Zuschauern gefallen. Viel Spaß beim Betrachtenden Zuschauern gefallen!
