v51

latestOpenAPI 3.1.0proprietaryraw.githubusercontent.com2026-07-21411711.1 MB

Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

post/v1/tokenize

Headers

X-Client-Namestring

The name of the project that is making the request.

Request body

textstring required

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

modelstring required

The input will be tokenized by the tokenizer that is used by this model.

Response

tokensinteger[] required

An array of tokens, where each token is an integer.

token_stringsstring[] required