--- title: "Create or Upsert Chunk or Chunks" method: POST path: "/api/chunk" tags: ["Chunk"] --- # Create or Upsert Chunk or Chunks `POST /api/chunk` Create new chunk(s). If the chunk has the same tracking_id as an existing chunk, the request will fail. Once a chunk is created, it can be searched for using the search endpoint. If uploading in bulk, the maximum amount of chunks that can be uploaded at once is 120 chunks. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. ## Headers - `TR-Dataset` string, uuid, required ## Request body - union - CreateSingleChunkReqPayload — Request payload for creating a new chunk - `chunk_html` string, nullable — HTML content of the chunk. This can also be plaintext. The innerText of the HTML will be used to create the embedding vector. The point of using HTML is for convienience, as some users have applications where users submit HTML content. - `convert_html_to_text` boolean, nullable — Convert HTML to raw text before processing to avoid adding noise to the vector embeddings. By default this is true. If you are using HTML content that you want to be included in the vector embeddings, set this to false. - `fulltext_boost` FullTextBoost — Boost the presence of certain tokens for fulltext (SPLADE) and keyword (BM25) search. I.e. boosting title phrases to priortize title matches or making sure that the listing for AirBNB itself ranks higher than companies who make software for AirBNB hosts by boosting the in-document-frequency of the AirBNB token (AKA word) for its official listing. Conceptually it multiples the in-document-importance second value in the tuples of the SPLADE or BM25 sparse vector of the chunk_html innerText for all tokens present in the boost phrase by the boost factor like so: (token, in-document-importance) -> (token, in-document-importance*boost_factor). - `boost_factor` number, double, required — Amount to multiplicatevly increase the frequency of the tokens in the phrase by - `phrase` string, required — The phrase to boost in the fulltext document frequency index - `fulltext_content` string, nullable — If fulltext_content is present, it will be used for creating the fulltext and bm25 sparse vectors instead of the innerText `chunk_html`. `chunk_html` will still be the only thing stored and used for semantic functionality unless the corresponding `semantic_content` field is defined. `chunk_html` must still be present for the chunk to be created properly. - `group_ids` string[], nullable — Group ids are the Trieve generated ids of the groups that the chunk should be placed into. This is useful for when you want to create a chunk and add it to a group or multiple groups in one request. Groups with these Trieve generated ids must be created first, it cannot be arbitrarily created through this route. - `group_tracking_ids` string[], nullable — Group tracking_ids are the user-assigned tracking_ids of the groups that the chunk should be placed into. This is useful for when you want to create a chunk and add it to a group or multiple groups in one request. If a group with the tracking_id does not exist, it will be created. - `high_priority` boolean, nullable — High Priority allows you to place this chunk into a priority queue with its own ingestion workers. Can only be used by users with a Custom Pro plan. - `image_urls` string[], nullable — Image urls are a list of urls to images that are associated with the chunk. This is useful for when you want to associate images with a chunk. - `link` string, nullable — Link to the chunk. This can also be any string. Frequently, this is a link to the source of the chunk. The link value will not affect the embedding creation. - `location` GeoInfo — Location that you want to use as the center of the search. - `lat` union, required - integer - number, double - `lon` union, required - integer - number, double - `metadata` unknown - `num_value` number, double, nullable — Num value is an arbitrary numerical value that can be used to filter chunks. This is useful for when you want to filter chunks by numerical value. There is no performance hit for filtering on num_value. - `semantic_boost` SemanticBoost — Semantic boosting moves the dense vector of the chunk in the direction of the distance phrase for semantic search. I.e. you can force a cluster by moving every chunk for a PDF closer to its title or push a chunk with a chunk_html of "iphone" 25% closer to the term "flagship" by using the distance phrase "flagship" and a distance factor of 0.25. Conceptually it's drawing a line (euclidean/L2 distance) between the vector for the innerText of the chunk_html and distance_phrase then moving the vector of the chunk_html distance_factor*L2Distance closer to or away from the distance_phrase point along the line between the two points. - `distance_factor` number, float, required — Arbitrary float (positive or negative) specifying the multiplicate factor to apply before summing the phrase vector with the chunk_html embedding vector - `phrase` string, required — Terms to embed in order to create the vector which is weighted summed with the chunk_html embedding vector - `semantic_content` string, nullable — If semantic_content is present, it will be used for creating semantic embeddings instead of the innerText `chunk_html`. `chunk_html` will still be the only thing stored and used for fulltext functionality unless the corresponding `fulltext_content` field is defined. `chunk_html` must still be present for the chunk to be created properly. - `split_avg` boolean, nullable — Split avg is a boolean which tells the server to split the text in the chunk_html into smaller chunks and average their resulting vectors. This is useful for when you want to create a chunk from a large piece of text and want to split it into smaller chunks to create a more fuzzy average dense vector. The sparse vector will be generated normally with no averaging. By default this is false. - `tag_set` string[], nullable — Tag set is a list of tags. This can be used to filter chunks by tag. Unlike with metadata filtering, HNSW indices will exist for each tag such that there is not a performance hit for filtering on them. - `time_stamp` string, nullable — Time_stamp should be an ISO 8601 combined date and time without timezone. It is used for time window filtering and recency-biasing search results. - `tracking_id` string, nullable — Tracking_id is a string which can be used to identify a chunk. This is useful for when you are coordinating with an external system and want to use the tracking_id to identify the chunk. - `upsert_by_tracking_id` boolean, nullable — Upsert when a chunk with the same tracking_id exists. By default this is false, and chunks will be ignored if another with the same tracking_id exists. If this is true, the chunk will be updated if a chunk with the same tracking_id exists. - `weight` number, double, nullable — Weight is a float which can be used to bias search results. This is useful for when you want to bias search results for a chunk. The magnitude only matters relative to other chunks in the chunk's dataset dataset. - ChunkReqPayload[] - `chunk_html` string, nullable — HTML content of the chunk. This can also be plaintext. The innerText of the HTML will be used to create the embedding vector. The point of using HTML is for convienience, as some users have applications where users submit HTML content. - `convert_html_to_text` boolean, nullable — Convert HTML to raw text before processing to avoid adding noise to the vector embeddings. By default this is true. If you are using HTML content that you want to be included in the vector embeddings, set this to false. - `fulltext_boost` FullTextBoost — Boost the presence of certain tokens for fulltext (SPLADE) and keyword (BM25) search. I.e. boosting title phrases to priortize title matches or making sure that the listing for AirBNB itself ranks higher than companies who make software for AirBNB hosts by boosting the in-document-frequency of the AirBNB token (AKA word) for its official listing. Conceptually it multiples the in-document-importance second value in the tuples of the SPLADE or BM25 sparse vector of the chunk_html innerText for all tokens present in the boost phrase by the boost factor like so: (token, in-document-importance) -> (token, in-document-importance*boost_factor). - `boost_factor` number, double, required — Amount to multiplicatevly increase the frequency of the tokens in the phrase by - `phrase` string, required — The phrase to boost in the fulltext document frequency index - `fulltext_content` string, nullable — If fulltext_content is present, it will be used for creating the fulltext and bm25 sparse vectors instead of the innerText `chunk_html`. `chunk_html` will still be the only thing stored and used for semantic functionality unless the corresponding `semantic_content` field is defined. `chunk_html` must still be present for the chunk to be created properly. - `group_ids` string[], nullable — Group ids are the Trieve generated ids of the groups that the chunk should be placed into. This is useful for when you want to create a chunk and add it to a group or multiple groups in one request. Groups with these Trieve generated ids must be created first, it cannot be arbitrarily created through this route. - `group_tracking_ids` string[], nullable — Group tracking_ids are the user-assigned tracking_ids of the groups that the chunk should be placed into. This is useful for when you want to create a chunk and add it to a group or multiple groups in one request. If a group with the tracking_id does not exist, it will be created. - `high_priority` boolean, nullable — High Priority allows you to place this chunk into a priority queue with its own ingestion workers. Can only be used by users with a Custom Pro plan. - `image_urls` string[], nullable — Image urls are a list of urls to images that are associated with the chunk. This is useful for when you want to associate images with a chunk. - `link` string, nullable — Link to the chunk. This can also be any string. Frequently, this is a link to the source of the chunk. The link value will not affect the embedding creation. - `location` GeoInfo — Location that you want to use as the center of the search. - `lat` union, required - integer - number, double - `lon` union, required - integer - number, double - `metadata` unknown - `num_value` number, double, nullable — Num value is an arbitrary numerical value that can be used to filter chunks. This is useful for when you want to filter chunks by numerical value. There is no performance hit for filtering on num_value. - `semantic_boost` SemanticBoost — Semantic boosting moves the dense vector of the chunk in the direction of the distance phrase for semantic search. I.e. you can force a cluster by moving every chunk for a PDF closer to its title or push a chunk with a chunk_html of "iphone" 25% closer to the term "flagship" by using the distance phrase "flagship" and a distance factor of 0.25. Conceptually it's drawing a line (euclidean/L2 distance) between the vector for the innerText of the chunk_html and distance_phrase then moving the vector of the chunk_html distance_factor*L2Distance closer to or away from the distance_phrase point along the line between the two points. - `distance_factor` number, float, required — Arbitrary float (positive or negative) specifying the multiplicate factor to apply before summing the phrase vector with the chunk_html embedding vector - `phrase` string, required — Terms to embed in order to create the vector which is weighted summed with the chunk_html embedding vector - `semantic_content` string, nullable — If semantic_content is present, it will be used for creating semantic embeddings instead of the innerText `chunk_html`. `chunk_html` will still be the only thing stored and used for fulltext functionality unless the corresponding `fulltext_content` field is defined. `chunk_html` must still be present for the chunk to be created properly. - `split_avg` boolean, nullable — Split avg is a boolean which tells the server to split the text in the chunk_html into smaller chunks and average their resulting vectors. This is useful for when you want to create a chunk from a large piece of text and want to split it into smaller chunks to create a more fuzzy average dense vector. The sparse vector will be generated normally with no averaging. By default this is false. - `tag_set` string[], nullable — Tag set is a list of tags. This can be used to filter chunks by tag. Unlike with metadata filtering, HNSW indices will exist for each tag such that there is not a performance hit for filtering on them. - `time_stamp` string, nullable — Time_stamp should be an ISO 8601 combined date and time without timezone. It is used for time window filtering and recency-biasing search results. - `tracking_id` string, nullable — Tracking_id is a string which can be used to identify a chunk. This is useful for when you are coordinating with an external system and want to use the tracking_id to identify the chunk. - `upsert_by_tracking_id` boolean, nullable — Upsert when a chunk with the same tracking_id exists. By default this is false, and chunks will be ignored if another with the same tracking_id exists. If this is true, the chunk will be updated if a chunk with the same tracking_id exists. - `weight` number, double, nullable — Weight is a float which can be used to bias search results. This is useful for when you want to bias search results for a chunk. The magnitude only matters relative to other chunks in the chunk's dataset dataset. ## Response `200` JSON response payload containing the created chunk - union - SingleQueuedChunkResponse - `chunk_metadata` ChunkMetadata, required - `chunk_html` string, nullable — HTML content of the chunk, can also be an arbitrary string which is not HTML - `created_at` string, date-time, required — Timestamp of the creation of the chunk - `dataset_id` string, uuid, required — ID of the dataset which the chunk belongs to - `id` string, uuid, required — Unique identifier of the chunk, auto-generated uuid created by Trieve - `image_urls` string[], nullable — Image URLs of the chunk, can be any list of strings. Used for image search and RAG. - `link` string, nullable — Link to the chunk, should be a URL - `location` GeoInfo — Location that you want to use as the center of the search. - `lat` union, required - integer - number, double - `lon` union, required - integer - number, double - `metadata` unknown - `num_value` number, double, nullable — Numeric value of the chunk, can be any float. Can represent the most relevant numeric value of the chunk, such as a price, quantity in stock, rating, etc. - `tag_set` string[], nullable — Tag set of the chunk, can be any list of strings. Used for tag-filtered searches. - `time_stamp` string, date-time, nullable — Timestamp of the chunk, can be any timestamp. Specified by the user. - `tracking_id` string, nullable — Tracking ID of the chunk, can be any string, determined by the user. Tracking ID's are unique identifiers for chunks within a dataset. They are designed to match the unique identifier of the chunk in the user's system. - `updated_at` string, date-time, required — Timestamp of the last update of the chunk - `weight` number, double, required — Weight of the chunk, can be any float. Used as a multiplier on a chunk's relevance score for ranking purposes. - BatchQueuedChunkResponse - `chunk_metadata` ChunkMetadata[], required - `chunk_html` string, nullable — HTML content of the chunk, can also be an arbitrary string which is not HTML - `created_at` string, date-time, required — Timestamp of the creation of the chunk - `dataset_id` string, uuid, required — ID of the dataset which the chunk belongs to - `id` string, uuid, required — Unique identifier of the chunk, auto-generated uuid created by Trieve - `image_urls` string[], nullable — Image URLs of the chunk, can be any list of strings. Used for image search and RAG. - `link` string, nullable — Link to the chunk, should be a URL - `location` GeoInfo — Location that you want to use as the center of the search. - `lat` union, required - integer - number, double - `lon` union, required - integer - number, double - `metadata` unknown - `num_value` number, double, nullable — Numeric value of the chunk, can be any float. Can represent the most relevant numeric value of the chunk, such as a price, quantity in stock, rating, etc. - `tag_set` string[], nullable — Tag set of the chunk, can be any list of strings. Used for tag-filtered searches. - `time_stamp` string, date-time, nullable — Timestamp of the chunk, can be any timestamp. Specified by the user. - `tracking_id` string, nullable — Tracking ID of the chunk, can be any string, determined by the user. Tracking ID's are unique identifiers for chunks within a dataset. They are designed to match the unique identifier of the chunk in the user's system. - `updated_at` string, date-time, required — Timestamp of the last update of the chunk - `weight` number, double, required — Weight of the chunk, can be any float. Used as a multiplier on a chunk's relevance score for ranking purposes. ## Other responses - `400` — Error typically due to deserialization issues - `413` — Error when more than 120 chunks are provided in bulk - `426` — Error when upgrade is needed to process more chunks --- [API](https://skmtc.net/devflowinc/apis/trieve-api.md) · [All operations](https://skmtc.net/devflowinc/apis/trieve-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/devflowinc/trieve-api/revisions/84583e7c9fc1/schema)