Provides incremental decoding of token IDs as they are generated, yielding decoded text chunks as soon as they are available.

Unlike batch decoding, streaming decode is designed for use with autoregressive generation — tokens arrive one at a time and the decoder needs to handle multi-byte sequences (e.g., UTF-8 characters split across token boundaries) and byte-fallback tokens gracefully.

The decoder internally buffers tokens until it can produce a valid UTF-8 string chunk, then yields that chunk and advances its internal state. This means individual calls to step() may return None when the current token completes a partial sequence that cannot yet be decoded.

>>> from tokenizers import Tokenizer
>>> from tokenizers.decoders import DecodeStream
>>> tokenizer = Tokenizer.from_pretrained("gpt2")
>>> stream = DecodeStream(skip_special_tokens=True)
>>> # Simulate streaming token-by-token generation
>>> token_ids = tokenizer.encode("Hello, streaming world!").ids
>>> for token_id in token_ids:
...     chunk = stream.step(tokenizer, token_id)
...     if chunk is not None:
...         print(chunk, end="", flush=True)

BPEDecoder Decoder

>>> from tokenizers.decoders import BPEDecoder
>>> decoder = BPEDecoder()
>>> decoder.decode(["Hello</w>", "world</w>"])
'Hello world'

ByteFallback Decoder

ByteFallback is a decoder that handles tokens representing raw bytes in the <0xNN> format (e.g., <0x61> for the byte 0x61 = 'a'). It converts such tokens to their corresponding bytes and attempts to decode the resulting byte sequence as UTF-8. This is used in LLaMA/SentencePiece models that use byte fallback for unknown characters. Inconvertible byte tokens are replaced with the Unicode replacement character (U+FFFD).

>>> from tokenizers.decoders import ByteFallback, Fuse, Sequence
>>> decoder = Sequence([ByteFallback(), Fuse()])
>>> decoder.decode(["<0x48>", "<0x65>", "<0x6C>", "<0x6C>", "<0x6F>"])
'Hello'

ByteLevel Decoder

This decoder is to be used in tandem with the ByteLevel pre-tokenizer. It reverses the byte-to-unicode mapping applied during pre-tokenization, converting the special Unicode characters back into the original bytes to reconstruct the original string.

>>> from tokenizers.decoders import ByteLevel
>>> decoder = ByteLevel()
>>> decoder.decode(["ĠHello", "Ġworld"])
' Hello world'

CTC Decoder

>>> from tokenizers.decoders import CTC
>>> decoder = CTC()
>>> decoder.decode(["h", "e", "e", "<pad>", "l", "l", "o", "|", "w", "o", "r", "l", "d"])
'hello world'

Fuse Decoder

Fuse simply concatenates every token into a single string without any separator. This is typically the last step in a decoder chain when other decoders need to operate on individual tokens before they are joined together.

>>> from tokenizers.decoders import Fuse
>>> decoder = Fuse()
>>> decoder.decode(["Hello", ",", " ", "world", "!"])
'Hello, world!'

Metaspace Decoder

>>> from tokenizers.decoders import Metaspace
>>> decoder = Metaspace()
>>> decoder.decode(["▁Hello", "▁my", "▁friend"])
'Hello my friend'

Replace Decoder

This decoder is to be used in tandem with the Replace normalizer or a similar replace operation. It reverses a string replacement by substituting the replacement content back with the original pattern.

>>> from tokenizers.decoders import Replace
>>> decoder = Replace("▁", " ")
>>> decoder.decode(["▁Hello", "▁world"])
' Hello world'

Sequence Decoder

Chains multiple decoders together, applying them in order. Each decoder in the sequence processes the output of the previous one, allowing complex decoding pipelines to be built from simpler components.

>>> from tokenizers.decoders import ByteFallback, Fuse, Metaspace, Sequence
>>> decoder = Sequence([ByteFallback(), Fuse(), Metaspace()])
>>> decoder.decode(["▁Hello", "▁world"])
'Hello world'

Strip Decoder

Strips a given number of occurrences of a character from the left and/or right side of each token. This is useful for removing padding characters or special prefix/suffix markers added during tokenization.

>>> from tokenizers.decoders import Strip
>>> decoder = Strip(content="▁", left=1)
>>> decoder.decode(["▁Hello", "▁world"])
'Hello world'

WordPiece Decoder

>>> from tokenizers.decoders import WordPiece
>>> decoder = WordPiece()
>>> decoder.decode(["Hello", ",", "##world", "!"])
'Hello, world!'