Create a corpus

Creates a corpus to store and manage your documents. A corpus is a container for documents and their associated metadata. You can set the corpus key, name, description, encoder, and filter attributes.

Corpus object

The key property is required and uniquely identifies the corpus. The name property is optional and defaults to the value of key. The optional description property provides additional information about the corpus. You can choose a key that follows a naming convention of your choice. An identifiable key makes a corpus easier to manage and reference in your application.

Use the queries_are_answers and documents_are_questions boolean properties to treat queries or documents in the corpus as questions or answers. These settings affect the semantics of the encoder at query time and indexing time.

Add metadata as filter attributes

Define metadata fields with the filter_attributes object when you create a corpus with this endpoint or the Vectara Console. The corpus then supports filtering on specific metadata attributes at the document level or the part level.

Filter attributes attach metadata to your data at the document (doc) or part level. You can use this metadata later in filter expressions to narrow the scope of your queries. A filter attribute must specify a unique name (up to 64 characters long) and a level of doc or part. At indexing time, the platform extracts metadata with this name and makes it available to filter expressions. Learn more

Doc and part filter levels

The doc attribute applies to the entire document. Use this for metadata that is consistent across the whole document, such as author, publication date, and document ID.

The part attribute applies to specific sections or chunks within a document. Use this for metadata that varies within different parts of the document, such as sections, page numbers, and sentiment scores.

If indexed is true, the platform builds an index on the extracted values. This index improves the performance of filter expressions that use the attribute.

Filter attributes must specify a type. The platform validates the type when documents are indexed. The four supported types are:

• integer: stores signed whole-number values up to eight bytes in length.
• real: stores floating point values in [IEEE 754 8-byte format].
• text: stores textual strings in [UTF-8 encoding].
• boolean: stores true/false values.

After you define filter attributes, you can use them within your queries. For example:

• Document-level attribute: doc.publication_year > 2020
• Part-level attribute: part.sentiment_score > 0.7

Custom dimensions

Custom dimensions add user-defined values to your data, in addition to what the platform automatically extracts and stores from the text. For example, upvotes can be a custom dimension. For an example, see Add custom dimensions to boost content.

{
  "key": "my-corpus",
  "name": "My corpus",
  "description": "Documents with important information for my prompt.",
  "encoder_id": "enc_1",
  "encoder_name": "boomerang-2023-q3",
  "filter_attributes": [
    {
      "name": "Title",
      "level": "document",
      "description": "The title of the document.",
      "type": "text"
    }
  ],
  "custom_dimensions": [
    {
      "name": "importance",
      "description": "Product importance."
    }
  ]
}

{
  "id": "crp_1",
  "key": "my-corpus",
  "name": "Vectara Content",
  "description": "Description about the Vectara Content corpus.",
  "encoder_name": "boomerang-2023-q3",
  "filter_attributes": [
    {
      "name": "Title",
      "level": "document",
      "description": "The title of the document.",
      "type": "text"
    }
  ],
  "custom_dimensions": [
    {
      "name": "importance",
      "description": "Product importance."
    }
  ],
  "limits": {
    "used_docs": 8,
    "used_parts": 3748,
    "used_bytes": 795790,
    "used_characters": 1839847,
    "max_metadata_bytes": 10485760
  }
}