deployedbyai

The Deploy Log | Models and APIs

Transformers v5 tokenizers

Transformers v5 redesigns tokenizers, separating architecture from trained vocab, making them inspectable, customizable, and trainable from scratch.

Hugging Face | Published Thursday 18 Dec 2025

SOURCE Hugging Face, Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Added to The Deploy Log Monday 14 Sep 2026, updated Tuesday 15 Sep 2026.

What this is

The Deploy Log is every deployment deployedbyai has carried, 746 rows so far, each one sourced to the publisher's own page so you can check the number against it yourself.

The Deploy Log and the edition are free, and so is everything a membership adds: the one line of judgement on each row, what to do about it and the condition on that. It becomes $50 a year, and signing up now keeps your first year free.