---
title: "Tokenize"
method: POST
path: "/v1/tokenize"
---

# Tokenize

`POST /v1/tokenize`

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

## Headers

- `X-Client-Name` string

## Request body

- object
  - `text` string, required — The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
  - `model` string, required — The input will be tokenized by the tokenizer that is used by this model.

## Response `200`

OK

- object
  - `tokens` integer[], required — An array of tokens, where each token is an integer.
  - `token_strings` string[], required
  - `meta` ApiMeta
    - `api_version` object
      - `version` string, required
      - `is_deprecated` boolean
      - `is_experimental` boolean
    - `billed_units` object
      - `images` number — The number of billed images.
      - `input_tokens` number — The number of billed input tokens.
      - `image_tokens` number — The number of billed image tokens.
      - `output_tokens` number — The number of billed output tokens.
      - `search_units` number — The number of billed search units.
      - `classifications` number — The number of billed classifications units.
    - `tokens` object
      - `input_tokens` number — The number of tokens used as input to the model.
      - `output_tokens` number — The number of tokens produced by the model.
    - `cached_tokens` number — The number of prompt tokens that hit the inference cache.
    - `warnings` string[]

## Other responses

- `400` — This error is returned when the request is not well formed. This could be because: - JSON is invalid - The request is missing required fields - The request contains an invalid combination of fields
- `401` — This error indicates that the operation attempted to be performed is not allowed. This could be because: - The api token is invalid - The user does not have the necessary permissions
- `403` — This error indicates that the operation attempted to be performed is not allowed. This could be because: - The api token is invalid - The user does not have the necessary permissions
- `404` — This error is returned when a resource is not found. This could be because: - The endpoint does not exist - The resource does not exist eg model id, dataset id
- `422` — This error is returned when the request is not well formed. This could be because: - JSON is invalid - The request is missing required fields - The request contains an invalid combination of fields
- `429` — Too many requests
- `498` — This error is returned when a request or response contains a deny-listed token.
- `499` — This error is returned when a request is cancelled by the user.
- `500` — This error is returned when an uncategorised internal server error occurs.
- `501` — This error is returned when the requested feature is not implemented.
- `503` — This error is returned when the service is unavailable. This could be due to: - Too many users trying to access the service at the same time
- `504` — This error is returned when a request to the server times out. This could be due to: - An internal services taking too long to respond

---

[API](https://skmtc.net/cohere-ai/apis/api-reference.md) · [All operations](https://skmtc.net/cohere-ai/apis/api-reference/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/cohere-ai/api-reference/versions/dbed69eda210/schema)