Learn how to use the 🤗 Tokenizers library to build your own tokenizer, train it, then how to use it in the 🤗 Transformers library.

This video is part of the Hugging Face course: [ Ссылка ]
Open in colab to run the code samples:
[ Ссылка ]

Related videos:
- Training a new tokenizer: [ Ссылка ]
- Byte Pair Encoding Tokenization: [ Ссылка ]
- Unigram Tokenization: [ Ссылка ]
- WordPiece Tokenization: [ Ссылка ]

Don't have a Hugging Face account? Join now: [ Ссылка ]
Have a question? Checkout the forums: [ Ссылка ]
Subscribe to our newsletter: [ Ссылка ]

свернуть