---
title: "Tokenize inputs"
method: POST
path: "/tokenize"
tags: ["Text Embeddings Inference"]
---

# Tokenize inputs

`POST /tokenize`

## Request body

- TokenizeRequest
  - `add_special_tokens` boolean
  - `inputs` union, required
    - string
    - string[]
  - `prompt_name` string, nullable — The name of the prompt that should be used by for encoding. If not set, no prompt will be applied. Must be a key in the `sentence-transformers` configuration `prompts` dictionary. For example if ``prompt_name`` is "query" and the ``prompts`` is {"query": "query: ", ...}, then the sentence "What is the capital of France?" will be encoded as "query: What is the capital of France?" because the prompt text will be prepended before any text to encode.

## Response `200`

Tokenized ids

- array[]
  - SimpleToken[]
    - `id` integer, required
    - `special` boolean, required
    - `start` integer, nullable
    - `stop` integer, nullable
    - `text` string, required

## Other responses

- `400` — Batch is empty
- `413` — Batch size error
- `422` — Tokenization error

---

[API](https://skmtc.net/huggingface/apis/text-embeddings-inference.md) · [All operations](https://skmtc.net/huggingface/apis/text-embeddings-inference/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/huggingface/text-embeddings-inference/versions/f4c15add6d2b/schema)