{"id":22582,"date":"2004-08-22T14:01:28","date_gmt":"2004-08-22T14:01:28","guid":{"rendered":"https:\/\/scannn.com\/marcelroed-gigatoken-language-model-tokenization-at-gb-s-%c2%b7-github\/"},"modified":"2004-08-22T14:01:28","modified_gmt":"2004-08-22T14:01:28","slug":"marcelroed-gigatoken-language-model-tokenization-at-gb-s-%c2%b7-github","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/marcelroed-gigatoken-language-model-tokenization-at-gb-s-%c2%b7-github\/","title":{"rendered":"marcelroed\/gigatoken: Language model tokenization at GB\/s \u00b7 GitHub"},"content":{"rendered":"\n
\n
\n

~1000x faster than HuggingFace’s tokenizers, drop-in replacement.<\/p>\n

Tokenize your text data at GB\/s!<\/em><\/p>\n

<\/a><\/p>\n

Note that both HF tokenizers and tiktoken are already running multithreaded Rust!<\/p>\n<\/div>\n

Gigatoken is the fastest tokenizer for language modeling.
\nIt supports a wide range of CPU hardware, and nearly all commonly used tokenizers.
\nSee the Benchmarks<\/a> section for detailed throughput numbers across tokenizers and CPUs.<\/p>\n

Gigatoken can be used with its own API, or in compatibility mode with HuggingFace Tokenizers or Tiktoken.<\/p>\n

\n

Compatibility Mode (Easiest)<\/h3>\n

<\/svg><\/a><\/div>\n

\n
import<\/span> gigatoken<\/span> as<\/span> gt<\/span>\n\n# Minimum change from existing HuggingFace tokenizers usage (compatibility mode)<\/span>\nhf_tokenizer<\/span> =<\/span> ...\ntokenizer<\/span> =<\/span> gt<\/span>.Tokenizer<\/span>(hf_tokenizer<\/span>).as_hf<\/span>()\n\n# tokenizer can be used in the same contexts as hf_tokenizer<\/span>\ntokens<\/span> =<\/span> tokenizer<\/span>.encode_batch<\/span>([\"This is a test string\"<\/span>, \"And here is another\"<\/span>])\n\n# OR with tiktoken<\/span>\ntiktokenizer<\/span> =<\/span> ...\ntokenizer<\/span> =<\/span> gt<\/span>.Tokenizer<\/span>(tiktokenizer<\/span>).as_tiktoken<\/span>()\n\n# Now works like existing tiktoken tokenizers<\/span>\ntokens<\/span> =<\/span> tokenizer<\/span>.encode_batch<\/span>([\"This is a test string\"<\/span>, \"And here is another\"<\/span>])<\/pre>\n<\/div>\n
A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance.
\nYou can still expect way faster performance across the board, but not quite the 1000x you will get with the Gigatoken API.<\/p>\n
\n
import<\/span> gigatoken<\/span> as<\/span> gt<\/span>\n\ntokenizer<\/span> =<\/span> gt<\/span>.Tokenizer<\/span>(\"Qwen\/Qwen3-8B\"<\/span>)  # Accepts HF model names<\/span>\nfile_source<\/span> =<\/span> gt<\/span>.TextFileSource<\/span>([\"owt_train.txt\"<\/span>], separator<\/span>=<\/span>b\"<|endoftext|>\"<\/span>)\ntokens<\/span> =<\/span> tokenizer<\/span>.encode_files<\/span>(file_source<\/span>)<\/pre>\n<\/div>\n
Using the Gigatoken API lets the Rust implementation read data directly, and skips as much overhead as possible while allowing for maximum parallelism.
\nKeep in mind that passing Python data structures through this API still incurs the overhead of reading from Python.<\/p>\n
\nEncoding throughput on owt_train.txt (11.9 GB) \u2014 AMD EPYC 9565 72-Core Processor x 2 sockets (144 cores)<\/b><\/summary>\n
<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Tokenizer<\/th>\ngigatoken<\/th>\nHF tokenizers<\/th>\ntiktoken<\/th>\nvs HF<\/th>\nvs tiktoken<\/th>\n<\/tr>\n<\/thead>\n
GPT-2<\/td>\n24.53 GB\/s<\/td>\n24.8 MB\/s<\/td>\n36.0 MB\/s<\/td>\n989\u00d7<\/td>\n681\u00d7<\/td>\n<\/tr>\n
Phi-4<\/td>\n24.00 GB\/s<\/td>\n29.9 MB\/s<\/td>\n\u2014<\/td>\n801\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GPT-OSS<\/td>\n23.96 GB\/s<\/td>\n49.7 MB\/s<\/td>\n42.8 MB\/s<\/td>\n482\u00d7<\/td>\n560\u00d7<\/td>\n<\/tr>\n
OLMo 2 \/ 3<\/td>\n23.06 GB\/s<\/td>\n27.7 MB\/s<\/td>\n\u2014<\/td>\n833\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Nemotron 3<\/td>\n22.79 GB\/s<\/td>\n49.4 MB\/s<\/td>\n\u2014<\/td>\n462\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 3<\/td>\n22.16 GB\/s<\/td>\n34.2 MB\/s<\/td>\n\u2014<\/td>\n648\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3 \/ 3.1 \/ 3.2<\/td>\n22.15 GB\/s<\/td>\n48.5 MB\/s<\/td>\n\u2014<\/td>\n457\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GLM 5<\/td>\n20.97 GB\/s<\/td>\n74.8 MB\/s<\/td>\n\u2014<\/td>\n280\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3.3<\/td>\n20.82 GB\/s<\/td>\n48.3 MB\/s<\/td>\n\u2014<\/td>\n431\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 4<\/td>\n20.77 GB\/s<\/td>\n72.7 MB\/s<\/td>\n\u2014<\/td>\n286\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GLM 4<\/td>\n20.61 GB\/s<\/td>\n72.3 MB\/s<\/td>\n\u2014<\/td>\n285\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Phi-4-mini<\/td>\n20.05 GB\/s<\/td>\n27.6 MB\/s<\/td>\n\u2014<\/td>\n726\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
DeepSeek V3 \/ R1 \/ V4<\/td>\n19.69 GB\/s<\/td>\n26.2 MB\/s<\/td>\n\u2014<\/td>\n750\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 2 \/ 2.5<\/td>\n19.12 GB\/s<\/td>\n27.7 MB\/s<\/td>\n\u2014<\/td>\n691\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Kimi K2<\/td>\n18.85 GB\/s<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 3.5 \/ 3.6<\/td>\n15.49 GB\/s<\/td>\n27.7 MB\/s<\/td>\n\u2014<\/td>\n558\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 4<\/td>\n4.82 GB\/s<\/td>\n334.1 MB\/s<\/td>\n\u2014<\/td>\n14\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
ModernBERT<\/td>\n4.18 GB\/s<\/td>\n26.9 MB\/s<\/td>\n\u2014<\/td>\n155\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Mistral 7B v0.3<\/td>\n3.57 GB\/s<\/td>\n354.7 MB\/s<\/td>\n\u2014<\/td>\n10\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
TinyLlama \/ Phi-3 (Llama 2)<\/td>\n3.48 GB\/s<\/td>\n323.6 MB\/s<\/td>\n\u2014<\/td>\n11\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
CodeLlama<\/td>\n3.47 GB\/s<\/td>\n347.4 MB\/s<\/td>\n\u2014<\/td>\n10.0\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 3<\/td>\n3.43 GB\/s<\/td>\n357.2 MB\/s<\/td>\n\u2014<\/td>\n9.6\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 1<\/td>\n2.51 GB\/s<\/td>\n342.2 MB\/s<\/td>\n\u2014<\/td>\n7.3\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/markdown-accessiblity-table>
\n<\/details>\n
\nEncoding throughput on owt_train.txt (11.9 GB) \u2014 Apple M4 Max (16 cores)<\/b><\/summary>\n
<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Tokenizer<\/th>\ngigatoken<\/th>\nHF tokenizers<\/th>\ntiktoken<\/th>\nvs HF<\/th>\nvs tiktoken<\/th>\n<\/tr>\n<\/thead>\n
GPT-2<\/td>\n8.79 GB\/s<\/td>\n6.9 MB\/s<\/td>\n62.8 MB\/s<\/td>\n1,268\u00d7<\/td>\n140\u00d7<\/td>\n<\/tr>\n
Nemotron 3<\/td>\n7.82 GB\/s<\/td>\n10.9 MB\/s<\/td>\n\u2014<\/td>\n715\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Phi-4<\/td>\n7.76 GB\/s<\/td>\n7.7 MB\/s<\/td>\n\u2014<\/td>\n1,012\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3 \/ 3.1 \/ 3.2<\/td>\n7.60 GB\/s<\/td>\n11.2 MB\/s<\/td>\n\u2014<\/td>\n676\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
OLMo 2 \/ 3<\/td>\n7.56 GB\/s<\/td>\n5.8 MB\/s<\/td>\n\u2014<\/td>\n1,299\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3.3<\/td>\n7.50 GB\/s<\/td>\n15.7 MB\/s<\/td>\n\u2014<\/td>\n479\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Phi-4-mini<\/td>\n6.97 GB\/s<\/td>\n7.2 MB\/s<\/td>\n\u2014<\/td>\n964\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Kimi K2<\/td>\n6.88 GB\/s<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 4<\/td>\n6.81 GB\/s<\/td>\n11.6 MB\/s<\/td>\n\u2014<\/td>\n590\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 2 \/ 2.5<\/td>\n6.37 GB\/s<\/td>\n5.8 MB\/s<\/td>\n\u2014<\/td>\n1,105\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 3<\/td>\n6.36 GB\/s<\/td>\n6.9 MB\/s<\/td>\n\u2014<\/td>\n918\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 3.5 \/ 3.6<\/td>\n6.31 GB\/s<\/td>\n6.3 MB\/s<\/td>\n\u2014<\/td>\n994\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GPT-OSS<\/td>\n6.20 GB\/s<\/td>\n20.2 MB\/s<\/td>\n87.2 MB\/s<\/td>\n306\u00d7<\/td>\n71\u00d7<\/td>\n<\/tr>\n
GLM 4<\/td>\n6.17 GB\/s<\/td>\n15.8 MB\/s<\/td>\n\u2014<\/td>\n392\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
DeepSeek V3 \/ R1 \/ V4<\/td>\n5.68 GB\/s<\/td>\n7.2 MB\/s<\/td>\n\u2014<\/td>\n788\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GLM 5<\/td>\n5.55 GB\/s<\/td>\n12.2 MB\/s<\/td>\n\u2014<\/td>\n456\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
ModernBERT<\/td>\n2.64 GB\/s<\/td>\n5.8 MB\/s<\/td>\n\u2014<\/td>\n452\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Mistral 7B v0.3<\/td>\n1.99 GB\/s<\/td>\n95.1 MB\/s<\/td>\n\u2014<\/td>\n21\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 4<\/td>\n1.82 GB\/s<\/td>\n85.2 MB\/s<\/td>\n\u2014<\/td>\n21\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
CodeLlama<\/td>\n1.73 GB\/s<\/td>\n80.2 MB\/s<\/td>\n\u2014<\/td>\n22\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
TinyLlama \/ Phi-3 (Llama 2)<\/td>\n1.69 GB\/s<\/td>\n80.1 MB\/s<\/td>\n\u2014<\/td>\n21\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 1<\/td>\n1.42 GB\/s<\/td>\n85.7 MB\/s<\/td>\n\u2014<\/td>\n17\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 3<\/td>\n1.38 GB\/s<\/td>\n82.2 MB\/s<\/td>\n\u2014<\/td>\n17\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/markdown-accessiblity-table>
\n<\/details>\n
\nEncoding throughput on owt_train.txt (11.9 GB) \u2014 AMD Ryzen 7 9800X3D 8-Core Processor (16 cores)<\/b><\/summary>\n
<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Tokenizer<\/th>\ngigatoken<\/th>\nHF tokenizers<\/th>\ntiktoken<\/th>\nvs HF<\/th>\nvs tiktoken<\/th>\n<\/tr>\n<\/thead>\n
GPT-2<\/td>\n6.27 GB\/s<\/td>\n59.0 MB\/s<\/td>\n92.1 MB\/s<\/td>\n106\u00d7<\/td>\n68\u00d7<\/td>\n<\/tr>\n
Phi-4<\/td>\n6.09 GB\/s<\/td>\n55.4 MB\/s<\/td>\n\u2014<\/td>\n110\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
OLMo 2 \/ 3<\/td>\n6.06 GB\/s<\/td>\n55.4 MB\/s<\/td>\n\u2014<\/td>\n109\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Phi-4-mini<\/td>\n5.80 GB\/s<\/td>\n54.6 MB\/s<\/td>\n\u2014<\/td>\n106\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GPT-OSS<\/td>\n5.68 GB\/s<\/td>\n79.6 MB\/s<\/td>\n112.7 MB\/s<\/td>\n71\u00d7<\/td>\n50\u00d7<\/td>\n<\/tr>\n
Qwen 3<\/td>\n5.34 GB\/s<\/td>\n54.4 MB\/s<\/td>\n\u2014<\/td>\n98\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 2 \/ 2.5<\/td>\n5.30 GB\/s<\/td>\n51.7 MB\/s<\/td>\n\u2014<\/td>\n103\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3.3<\/td>\n5.26 GB\/s<\/td>\n79.9 MB\/s<\/td>\n\u2014<\/td>\n66\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 3 \/ 3.1 \/ 3.2<\/td>\n5.24 GB\/s<\/td>\n79.5 MB\/s<\/td>\n\u2014<\/td>\n66\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Kimi K2<\/td>\n5.23 GB\/s<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n\u2014<\/td>\n<\/tr>\n
Qwen 3.5 \/ 3.6<\/td>\n5.22 GB\/s<\/td>\n51.6 MB\/s<\/td>\n\u2014<\/td>\n101\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Nemotron 3<\/td>\n5.20 GB\/s<\/td>\n79.0 MB\/s<\/td>\n\u2014<\/td>\n66\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GLM 5<\/td>\n5.05 GB\/s<\/td>\n79.5 MB\/s<\/td>\n\u2014<\/td>\n63\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
GLM 4<\/td>\n5.04 GB\/s<\/td>\n79.5 MB\/s<\/td>\n\u2014<\/td>\n63\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Llama 4<\/td>\n5.03 GB\/s<\/td>\n78.2 MB\/s<\/td>\n\u2014<\/td>\n64\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
DeepSeek V3 \/ R1 \/ V4<\/td>\n4.21 GB\/s<\/td>\n51.6 MB\/s<\/td>\n\u2014<\/td>\n82\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
ModernBERT<\/td>\n2.84 GB\/s<\/td>\n52.1 MB\/s<\/td>\n\u2014<\/td>\n54\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Mistral 7B v0.3<\/td>\n1.47 GB\/s<\/td>\n91.6 MB\/s<\/td>\n\u2014<\/td>\n16\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 4<\/td>\n1.45 GB\/s<\/td>\n78.8 MB\/s<\/td>\n\u2014<\/td>\n18\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
CodeLlama<\/td>\n1.38 GB\/s<\/td>\n85.2 MB\/s<\/td>\n\u2014<\/td>\n16\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
TinyLlama \/ Phi-3 (Llama 2)<\/td>\n1.37 GB\/s<\/td>\n84.9 MB\/s<\/td>\n\u2014<\/td>\n16\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 1<\/td>\n1.14 GB\/s<\/td>\n84.9 MB\/s<\/td>\n\u2014<\/td>\n13\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n
Gemma 3<\/td>\n1.12 GB\/s<\/td>\n83.0 MB\/s<\/td>\n\u2014<\/td>\n13\u00d7<\/td>\n\u2014<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/markdown-accessiblity-table>
\n<\/details>\n
\nBenchmark details<\/b><\/summary>\n
OWT (openwebtext) was chosen because it’s roughly representative of the text you get after extraction from CommonCrawl documents.
\nGigatoken encodes the whole file un-split, and is thus doing more work than the other tokenizers to find the split boundaries and automatically parallelize.
\nHuggingFace tokenizers (encode_batch_fast<\/code>) gets the first 100 MB and tiktoken (encode_ordinary_batch<\/code>) the first 1 GB, both presplit on <|endoftext|><\/code>.
\nThis is fair because neither of the compared tokenizers do caching, meaning the speed is roughly uniform throughout processing.
\nTiktoken rows are currently only filled in for tokenizers with official support.<\/p>\n
The slowest rows are the SentencePiece-based tokenizers, which are not well optimized in Gigatoken.<\/p>\n
Each row is one distinct tokenizer (identical vocab\/merges\/pretokenizer), measured on a representative repo.
\nIf you don’t see your tokenizer here, it’s likely based on some existing one.
\nFor instance:<\/p>\n