GigaToken, a new software tool published on GitHub by developer Marcel Rød, claims to work roughly 1,000 times faster than the widely used HuggingFace tokenizers while doing the same job. The tool appeared on July 22, 2026, and works in three modes: its own native mode, a HuggingFace compatibility mode, and a Tiktoken compatibility mode. In HuggingFace compatibility mode, the results match HuggingFace exactly, though the tool runs somewhat slower than in its native mode while still beating the original. Source: GitHub

Tokenization is the process of breaking text into smaller pieces called tokens so that an AI model can read it. If you think of a model as a factory that builds understanding from text, tokenization is the delivery truck that keeps raw material flowing in. If the truck is slow, the whole factory waits.

The speed comes from building the tool in Rust, a programming language known for performance, rather than Python, which is easier to write but slower to run. Rød, a third-year PhD student at Stanford with 91 repositories on GitHub, describes several specific techniques: using the CPU's ability to handle multiple pieces of data at once, cutting down on decision points in the code, and remembering words it has already processed so it does not have to redo work. The tool runs on modern x86 and ARM processors, with Rød reporting consistent results across both. Source: GitHub

The benchmark numbers in the repository are striking. On an Apple M4 Max with 16 cores, GigaToken processed 1,887.23 million tokens per second. HuggingFace processed 1.40 million tokens per second on the same machine, making GigaToken 1,353 times faster. On a high-end AMD server system with 72 cores and 144 threads, GigaToken reached 5,564.94 million tokens per second. Source: GitHub

The tool fits into existing setups by accepting HuggingFace model names directly, such as 'Qwen/Qwen3-8B', and loading the right tokenizer automatically. It also includes a command-line tool called 'gigatoken bench' that lets users test and verify tokenization for a HuggingFace model without doing a full installation. Source: GitHub

The techniques Rød used are well-known in the world of performance engineering. Writing in a fast language instead of a slower one, reducing unnecessary work in the code, and caching results so they do not have to be recomputed are all standard approaches. Using them for tokenization makes sense because the amount of text that needs processing for large AI models is enormous.

For teams that build AI systems, a tool that produces identical results to HuggingFace but far faster could let them check the claims against their own data with little effort. The 'gigatoken bench' command makes that verification straightforward. Tokenization is more often a slowdown during data preparation than during the actual model use, so speeding it up matters most for organizations processing large amounts of text on powerful computers.

The broader story here is about a familiar trade-off: tools that are easy to use are not always the fastest. HuggingFace made machine learning workflows accessible and convenient, but that convenience carries a performance cost. A Rust-based tool that keeps the familiar interface while running much faster gives infrastructure teams a way to cut preprocessing time without learning an entirely new system. Whether GigaToken performs as well on the messy, varied data of real-world use is something developers will have to test for themselves.