Sign in

v1

latestOpenAPI 3.0.32026-07-2659199427.5 KB
Get tokens

Tokenization by MODEL_ID

The tokenization request for the pre-trained and custom embedding use cases and specified embedding modelId (model name) sends text to return results in formats supported by embedding models.

post/ai/tokenization/{MODEL_ID}

Headers

Authorization: Bearer ACCESS_TOKENstring

The authentication and authorization access token.

Content-Typestring
Example:application/json

application/json

Request body

Example request

{
  "batch": [
    {
      "text": "Mr. and Mrs. Dursley and O'\\''Malley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much"
    }
  ],
  "useCaseConfig": {
    "dataType": "passage"
  },
  "modelConfig": {
    "vectorQuantizationMethod": "min-max",
    "dimReductionSize": 256
  }
}

Response

OK

Example response

{
  "generatedTokens": [
    {
      "tokens": [
        "\"[CLS]\",                 \"query\",                 \":\",                 \"mr\",                 \".\",                 \"and\",                 \"mrs\",                 \".\",                 \"du\",                 \"##rs\",                 \"##ley\",                 \"and\",                 \"o\",                 \"'\",                 \"malley\",                 \",\",                 \"of\",                 \"number\",                 \"four\",                 \",\",                 \"pri\",                 \"##vet\",                 \"drive\",                 \",\",                 \"were\",                 \"proud\",                 \"to\",                 \"say\",                 \"that\",                 \"they\",                 \"were\",                 \"perfectly\",                 \"normal\",                 \",\",                 \"thank\",                 \"you\",                 \"very\",                 \"much\",                 \".\",                 \"[SEP]\""
      ]
    }
  ],
  "tokensUsed": {
    "promptTokens": 148,
    "completionTokens": 27,
    "totalTokens": 175
  }
}