--- title: "List destinations" method: GET path: "/destinations" tags: ["public_destinations", "public", "Destinations"] --- # List destinations `GET /destinations` ## Query parameters - `workspaceIds` string[] - `includeDeleted` boolean - `limit` integer - `offset` integer ## Response `200` Successful operation - DestinationsResponse - `previous` string - `next` string - `data` DestinationResponse[], required - `destinationId` string, UUID, required - `name` string, required - `destinationType` string, required - `definitionId` string, UUID, required - `workspaceId` string, UUID, required - `configuration` union, required — The values required to configure the destination. - DestinationGoogleSheets - `credentials` union, required — Authentication method to access Google Sheets - object - `auth_type` 'oauth2.0', required - `client_id` string, required — The Client ID of your Google Sheets developer application. - `client_secret` string, required — The Client Secret of your Google Sheets developer application. - `refresh_token` string, required — The token for obtaining new access token. - object - `auth_type` 'service', required - `service_account_info` string, required — Enter your service account key in JSON format. See the docs for more information on how to generate this key. - `spreadsheet_id` string, required — The link to your spreadsheet. See this guide for more details. - `destinationType` 'google-sheets', required - DestinationAstra — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Astra DB gives developers the APIs, real-time data and ecosystem integrations to put accurate RAG and Gen AI apps with fewer hallucinations in production. - `astra_db_app_token` string, required — The application token authorizes a user to connect to a specific Astra DB database. It is created when the user clicks the Generate Token button on the Overview tab of the Database page in the Astra UI. - `astra_db_endpoint` string, required — The endpoint specifies which Astra DB database queries are sent to. It can be copied from the Database Details section of the Overview tab of the Database page in the Astra UI. - `astra_db_keyspace` string, required — Keyspaces (or Namespaces) serve as containers for organizing data within a database. You can create a new keyspace uisng the Data Explorer tab in the Astra UI. The keyspace default_keyspace is created for you when you create a Vector Database in Astra DB. - `collection` string, required — Collections hold data. They are analagous to tables in traditional Cassandra terminology. This tool will create the collection with the provided name automatically if it does not already exist. Alternatively, you can create one thorugh the Data Explorer tab in the Astra UI. - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'astra', required - DestinationAwsDatalake - `aws_account_id` string — target aws account id - `bucket_name` string, required — The name of the S3 bucket. Read more here. - `bucket_prefix` string — S3 prefix - `credentials` union, required — Choose How to Authenticate to AWS. - object - `credentials_title` 'IAM Role', required — Name of the credentials - `role_arn` string, required — Will assume this role to write data to s3 - object - `aws_access_key_id` string, required — AWS User Access Key Id - `aws_secret_access_key` string, required — Secret Access Key - `credentials_title` 'IAM User', required — Name of the credentials - `format` union — Format of the data output. - object - `compression_codec` 'UNCOMPRESSED' | 'GZIP' — The compression algorithm used to compress data. - `format_type` 'JSONL', required - object - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'ZSTD' — The compression algorithm used to compress data. - `format_type` 'Parquet', required - `glue_catalog_float_as_decimal` boolean — Cast float/double as decimal(38,18). This can help achieve higher accuracy and represent numbers correctly as received from the source. - `lakeformation_database_default_tag_key` string — Add a default tag key to databases created by this destination - `lakeformation_database_default_tag_values` string — Add default values for the `Tag Key` to databases created by this destination. Comma separate for multiple values. - `lakeformation_database_name` string, required — The default database this destination will use to create tables in per stream. Can be changed per connection by customizing the namespace. - `lakeformation_governed_tables` boolean — Whether to create tables as LF governed tables. - `partitioning` 'NO PARTITIONING' | 'DATE' | 'YEAR' | 'MONTH' | 'DAY' | 'YEAR/MONTH' | 'YEAR/MONTH/DAY' — Partition data by cursor fields when a cursor field is a date - `region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `destinationType` 'aws-datalake', required - DestinationAzureBlobStorage - `azure_blob_storage_account_key` string — The Azure Blob Storage account key. If you set this value, you must not set the "Shared Access Signature", "Azure Tenant ID", "Azure Client ID", or "Azure Client Secret" fields. - `azure_blob_storage_account_name` string, required — The name of the Azure Blob Storage Account. Read more here. - `azure_blob_storage_container_name` string, required — The name of the Azure Blob Storage Container. Read more here. - `azure_blob_storage_endpoint_domain_name` string — This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example. - `azure_blob_storage_spill_size` integer — The amount of megabytes after which the connector should spill the records in a new blob object. Make sure to configure size greater than individual records. Enter 0 if not applicable. - `azure_client_id` string — The Azure Active Directory (Entra ID) client ID. Required for Entra ID authentication. - `azure_client_secret` string — The Azure Active Directory (Entra ID) client secret. Required for Entra ID authentication. - `azure_tenant_id` string — The Azure Active Directory (Entra ID) tenant ID. Required for Entra ID authentication. - `format` union, required — Format of the data output. - object - `flattening` 'No flattening' | 'Root level flattening', required - `format_type` 'CSV', required - object - `flattening` 'No flattening' | 'Root level flattening' - `format_type` 'JSONL', required - `shared_access_signature` string — A shared access signature (SAS) provides secure delegated access to resources in your storage account. Read more here. If you set this value, you must not set the "Azure Blob Storage Account Key", "Azure Tenant ID", "Azure Client ID", or "Azure Client Secret" fields. - `destinationType` 'azure-blob-storage', required - DestinationBigquery - `cdc_deletion_mode` 'Hard delete' | 'Soft delete' — Whether to execute CDC deletions as hard deletes (i.e. propagate source deletions to the destination), or soft deletes (i.e. leave a tombstone record in the destination). Defaults to hard deletes. - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `dataset_id` string, required — The default BigQuery Dataset ID that tables are replicated to if the source does not specify a namespace. Read more here. - `dataset_location` 'EU' | 'US' | 'africa-south1' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'europe-central2' | 'europe-north1' | 'europe-north2' | 'europe-southwest1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'europe-west8' | 'europe-west9' | 'europe-west10' | 'europe-west12' | 'me-central1' | 'me-central2' | 'me-west1' | 'northamerica-northeast1' | 'northamerica-northeast2' | 'northamerica-south1' | 'southamerica-east1' | 'southamerica-west1' | 'us-central1' | 'us-east1' | 'us-east4' | 'us-east5' | 'us-south1' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4', required — The location of the dataset. Warning: Changes made after creation will not be applied. Read more here. - `disable_type_dedupe` boolean — Write the legacy "raw tables" format, to enable backwards compatibility with older versions of this connector. - `loading_method` union — The way data will be uploaded to BigQuery. - object — Direct loading using batched SQL INSERT statements. This method uses the BigQuery driver to convert large INSERT statements into file uploads automatically. - `method` 'Standard', required - object — Writes large batches of records to a file, uploads the file to GCS, then uses COPY INTO to load your data into BigQuery. - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - … - `gcs_bucket_name` string, required — The name of the GCS bucket. Read more here. - `gcs_bucket_path` string, required — Directory under the GCS bucket where data will be written. - `keep_files_in_gcs-bucket` 'Delete all tmp files from GCS' | 'Keep all tmp files in GCS' — This upload method is supposed to temporary store records in GCS bucket. By this select you can chose if these records should be removed from GCS when migration has finished. The default "Delete all tmp files from GCS" value is used if not set explicitly. - `method` 'GCS Staging', required - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. Read more here. - `raw_data_dataset` string — Airbyte will use this dataset for various internal tables. In legacy raw tables mode, the raw tables will be stored in this dataset. Defaults to "airbyte_internal". - `destinationType` 'bigquery', required - DestinationClickhouse - `database` string, required — Name of the database. - `enable_json` boolean — Use the JSON type for Object fields. If disabled, the JSON will be converted to a string. - `host` string, required — Hostname of the database. - `password` string, required — Password associated with the username. - `port` string, required — HTTP port of the database. Default(s) HTTP: 8123 — HTTPS: 8443 - `protocol` 'http' | 'https', required — Protocol for the database connection string. - `record_window_size` integer — Warning: Tuning this parameter can impact the performances. The maximum number of records that should be written to a batch. The batch size limit is still limited to 70 Mb - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object — No ssh tunnel needed to connect to database - `tunnel_method` 'NO_TUNNEL', required - object — Connect through a jump server tunnel host using username and ssh key - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - object — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `username` string, required — Username to use to access the database. - `destinationType` 'clickhouse', required - DestinationConvex - `access_key` string, required — API access key used to send data to a Convex deployment. - `deployment_url` string, required — URL of the Convex deployment that is the destination - `destinationType` 'convex', required - DestinationCustomerIo - `credentials` object, required — Enter the site ID and API key to authenticate. - `apiKey` string, required — Enter your Customer IO API Key. - `siteId` string, required — Enter your Customer IO Site ID. - `object_storage_config` union - object - `storage_type` 'None', required - object - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `bucket_path` string, required — All files in the bucket will be prefixed by this. - `role_arn` string — The ARN of the AWS role to assume. Only usable in Airbyte Cloud. - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `storage_type` 'S3', required - `destinationType` 'customer-io', required - DestinationDatabricks - `accept_terms` boolean, required — You must agree to the Databricks JDBC Driver Terms & Conditions to use this connector. - `authentication` union, required — Authentication mechanism for Staging files and running queries - object - `auth_type` 'OAUTH', required - `client_id` string, required - `secret` string, required - object - `auth_type` 'BASIC', required - `personal_access_token` string, required - `database` string, required — The name of the unity catalog for the database - `hostname` string, required — Databricks Cluster Server Hostname. - `http_path` string, required — Databricks Cluster HTTP Path. - `port` string — Databricks Cluster Port. - `purge_staging_data` boolean — Default to 'true'. Switch it to 'false' for debugging purpose. - `raw_schema_override` string — The schema to write raw tables into (default: airbyte_internal) - `schema` string — The default schema tables are written. If not specified otherwise, the "default" will be used. - `destinationType` 'databricks', required - DestinationDeepset - `api_key` string, required — Your deepset cloud API key - `base_url` string — URL of deepset Cloud API (e.g. https://api.cloud.deepset.ai, https://api.us.deepset.ai, etc). Defaults to https://api.cloud.deepset.ai. - `retries` number — Number of times to retry an action before giving up. - `workspace` string, required — Name of workspace to which to sync the data. - `destinationType` 'deepset', required - DestinationDevNull - `test_destination` union, required — The type of destination to be used - object - `logging_config` union, required — Configurate how the messages are logged. - … - `test_destination_type` 'LOGGING', required - object - `test_destination_type` 'SILENT', required - object - `millis_per_record` integer, required — The number of milliseconds to wait between each record. - `test_destination_type` 'THROTTLED', required - object - `num_messages` integer, required — Number of messages after which to fail. - `test_destination_type` 'FAILING', required - `destinationType` 'dev-null', required - DestinationDuckdb - `destination_path` string, required — Path to the .duckdb file, or the text 'md:' to connect to MotherDuck. The file will be placed inside that local mount. For more information check out our docs - `motherduck_api_key` string — API key to use for authentication to a MotherDuck database. - `schema` string — Database schema name, default for duckdb is 'main'. - `destinationType` 'duckdb', required - DestinationDynamodb - `access_key_id` string, required — The access key id to access the DynamoDB. Airbyte requires Read and Write permissions to the DynamoDB. - `dynamodb_endpoint` string — This is your DynamoDB endpoint url.(if you are working with AWS DynamoDB, just leave empty). - `dynamodb_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the DynamoDB. - `dynamodb_table_name_prefix` string, required — The prefix to use when naming DynamoDB tables. - `secret_access_key` string, required — The corresponding secret to the access key id. - `destinationType` 'dynamodb', required - DestinationElasticsearch - `authenticationMethod` union — The type of authentication to be used - object — No authentication will be used - `method` 'none', required - object — Use a api key and secret combination to authenticate - `apiKeyId` string, required — The Key ID to used when accessing an enterprise Elasticsearch instance. - `apiKeySecret` string, required — The secret associated with the API Key ID. - `method` 'secret', required - object — Basic auth header with a username and password - `method` 'basic', required - `password` string, required — Basic auth password to access a secure Elasticsearch server - `username` string, required — Basic auth username to access a secure Elasticsearch server - `ca_certificate` string — CA certificate - `endpoint` string, required — The full url of the Elasticsearch server - `pathPrefix` string — The Path Prefix of the Elasticsearch server - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `upsert` boolean — If a primary key identifier is defined in the source, an upsert will be performed using the primary key value as the elasticsearch doc id. Does not support composite primary keys. - `destinationType` 'elasticsearch', required - DestinationFirebolt - `account` string, required — Firebolt account to login. - `client_id` string, required — Firebolt service account ID. - `client_secret` string, required — Firebolt secret, corresponding to the service account ID. - `database` string, required — The database to connect to. - `engine` string, required — Engine name to connect to. - `host` string — The host name of your Firebolt database. - `loading_method` union — Loading method used to select the way data will be uploaded to Firebolt - object - `method` 'SQL', required - object - `aws_key_id` string, required — AWS access key granting read and write access to S3. - `aws_key_secret` string, required — Corresponding secret part of the AWS Key - `method` 'S3', required - `s3_bucket` string, required — The name of the S3 bucket. - `s3_region` string, required — Region name of the S3 bucket. - `destinationType` 'firebolt', required - DestinationFirestore - `credentials_json` string — The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty. - `project_id` string, required — The GCP project ID for the project containing the target BigQuery dataset. - `destinationType` 'firestore', required - DestinationGcs - `credential` object, required — An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here. - `credential_type` 'HMAC_KEY', required - `hmac_key_access_id` string, required — When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. Read more here. - `hmac_key_secret` string, required — The corresponding secret for the access ID. It is a 40-character base-64 encoded string. Read more here. - `format` union, required — Output data format. One of the following formats must be selected - AVRO format, PARQUET format, CSV format, or JSONL format. - object - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - … - `format_type` 'Avro', required - object - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".csv.gz"). - … - `flattening` 'No flattening' | 'Root level flattening' — Whether the input JSON data should be normalized (flattened) in the output CSV. Please refer to docs for details. - `format_type` 'CSV', required - object - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - … - `format_type` 'JSONL', required - object - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `dictionary_encoding` boolean — Default: true. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `format_type` 'Parquet', required - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `gcs_bucket_name` string, required — You can find the bucket name in the App Engine Admin console Application Settings page, under the label Google Cloud Storage Bucket. Read more here. - `gcs_bucket_path` string, required — GCS Bucket Path string Subdirectory under the above bucket to sync the data into. - `gcs_bucket_region` 'northamerica-northeast1' | 'northamerica-northeast2' | 'us-central1' | 'us-east1' | 'us-east4' | 'us-west1' | 'us-west2' | 'us-west3' | 'us-west4' | 'southamerica-east1' | 'southamerica-west1' | 'europe-central2' | 'europe-north1' | 'europe-west1' | 'europe-west2' | 'europe-west3' | 'europe-west4' | 'europe-west6' | 'asia-east1' | 'asia-east2' | 'asia-northeast1' | 'asia-northeast2' | 'asia-northeast3' | 'asia-south1' | 'asia-south2' | 'asia-southeast1' | 'asia-southeast2' | 'australia-southeast1' | 'australia-southeast2' | 'asia' | 'eu' | 'us' | 'asia1' | 'eur4' | 'nam4' — Select a Region of the GCS Bucket. Read more here. - `destinationType` 'gcs', required - DestinationHubspot - `credentials` object, required — Choose how to authenticate to HubSpot. - `client_id` string, required — The Client ID of your HubSpot developer application. See the Hubspot docs if you need help finding this ID. - `client_secret` string, required — The client secret for your HubSpot developer application. See the Hubspot docs if you need help finding this secret. - `refresh_token` string, required — Refresh token to renew an expired access token. See the Hubspot docs if you need help finding this token. - `type` 'OAuth', required - `object_storage_config` union - object - `storage_type` 'None', required - object - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `bucket_path` string, required — All files in the bucket will be prefixed by this. - `role_arn` string — The ARN of the AWS role to assume. Only usable in Airbyte Cloud. - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `storage_type` 'S3', required - `destinationType` 'hubspot', required - DestinationMilvus — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Indexing configuration - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Zilliz Cloud) - … - object — Authenticate using username and password (suitable for self-managed Milvus clusters) - … - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - … - `collection` string, required — The collection to load data into - `db` string — The database to connect to - `host` string, required — The public endpoint of the Milvus instance. - `text_field` string — The field in the entity that contains the embedded text - `vector_field` string — The field in the entity that contains the vector - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'milvus', required - DestinationMongodb - `auth_type` union, required — Authorization type. - object — None. - `authorization` 'none', required - object — Login/Password. - `authorization` 'login/password', required - `password` string, required — Password associated with the username. - `username` string, required — Username to use to access the database. - `database` string, required — Name of the database. - `instance_type` union — MongoDb instance to connect to. For MongoDB Atlas and Replica Set TLS connection is used by default. - object - `host` string, required — The Host of a Mongo database to be replicated. - `instance` 'standalone', required - `port` integer, required — The Port of a Mongo database to be replicated. - `tls` boolean — Indicates whether TLS encryption protocol will be used to connect to MongoDB. It is recommended to use TLS connection if possible. For more information see documentation. - object - `instance` 'replica', required - `replica_set` string — A replica set name. - `server_addresses` string, required — The members of a replica set. Please specify `host`:`port` of each member seperated by comma. - object - `cluster_url` string, required — URL of a cluster to connect to. - `instance` 'atlas', required - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `destinationType` 'mongodb', required - DestinationMotherduck - `destination_path` string — Path to a .duckdb file or 'md:' to connect to a MotherDuck database. If 'md:' is specified without a database name, the default MotherDuck database name ('my_db') will be used. - `motherduck_api_key` string, required — API access token to use for authentication to a MotherDuck database. - `schema` string — Database schema name, defaults to 'main' if not specified. - `destinationType` 'motherduck', required - DestinationMssql - `database` string, required — The name of the MSSQL database. - `host` string, required — The host name of the MSSQL database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `load_type` union, required — Specifies the type of load mechanism (e.g., BULK, INSERT) and its associated configuration. - object — Configuration details for using the INSERT loading mechanism. - `load_type` 'INSERT', required - object — Configuration details for using the BULK loading mechanism. - `azure_blob_storage_account_key` string — The Azure blob storage account key. Mutually exclusive with a Shared Access Signature - `azure_blob_storage_account_name` string, required — The name of the Azure Blob Storage account. See: https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction#storage-accounts - `azure_blob_storage_container_name` string, required — The name of the Azure Blob Storage container. See: https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction#containers - `bulk_load_data_source` string, required — Specifies the external data source name configured in MSSQL, which references the Azure Blob container. See: https://learn.microsoft.com/sql/t-sql/statements/bulk-insert-transact-sql - `bulk_load_validate_values_pre_load` boolean — When enabled, Airbyte will validate all values before loading them into the destination table. This provides stronger data integrity guarantees but may significantly impact performance. - `load_type` 'BULK', required - `shared_access_signature` string — A shared access signature (SAS) provides secure delegated access to resources in your storage account. See: https://learn.microsoft.com/azure/storage/common/storage-sas-overview.Mutually exclusive with an account key - `password` string — The password associated with this username. - `port` integer, required — The port of the MSSQL database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl_method` union, required — The encryption method which is used to communicate with the database. - object — The data transfer will not be encrypted. - `name` 'unencrypted', required - object — Use the certificate provided by the server without verification. (For testing purposes only!) - `name` 'encrypted_trust_server_certificate', required - object — Verify and use the certificate provided by the server. - `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate. - `name` 'encrypted_verify_certificate', required - `trustStoreName` string — Specifies the name of the trust store. - `trustStorePassword` string — Specifies the password of the trust store. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object — No ssh tunnel needed to connect to database - `tunnel_method` 'NO_TUNNEL', required - object — Connect through a jump server tunnel host using username and ssh key - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - object — Connect through a jump server tunnel host using username and password authentication - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `user` string, required — The username which is used to access the database. - `destinationType` 'mssql', required - DestinationMssqlV2 - `database` string, required — The name of the MSSQL database. - `host` string, required — The host name of the MSSQL database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `load_type` union, required — Specifies the type of load mechanism (e.g., BULK, INSERT) and its associated configuration. - object — Configuration details for using the INSERT loading mechanism. - `load_type` 'INSERT', required - object — Configuration details for using the BULK loading mechanism. - `azure_blob_storage_account_name` string, required — The name of the Azure Blob Storage account. See: https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction#storage-accounts - `azure_blob_storage_container_name` string, required — The name of the Azure Blob Storage container. See: https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction#containers - `bulk_load_data_source` string, required — Specifies the external data source name configured in MSSQL, which references the Azure Blob container. See: https://learn.microsoft.com/sql/t-sql/statements/bulk-insert-transact-sql - `bulk_load_validate_values_pre_load` boolean — When enabled, Airbyte will validate all values before loading them into the destination table. This provides stronger data integrity guarantees but may significantly impact performance. - `load_type` 'BULK', required - `shared_access_signature` string, required — A shared access signature (SAS) provides secure delegated access to resources in your storage account. See: https://learn.microsoft.com/azure/storage/common/storage-sas-overview - `password` string — The password associated with this username. - `port` integer, required — The port of the MSSQL database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl_method` union, required — The encryption method which is used to communicate with the database. - object — The data transfer will not be encrypted. - `name` 'unencrypted', required - object — Use the certificate provided by the server without verification. (For testing purposes only!) - `name` 'encrypted_trust_server_certificate', required - object — Verify and use the certificate provided by the server. - `hostNameInCertificate` string — Specifies the host name of the server. The value of this property must match the subject property of the certificate. - `name` 'encrypted_verify_certificate', required - `trustStoreName` string — Specifies the name of the trust store. - `trustStorePassword` string — Specifies the password of the trust store. - `user` string, required — The username which is used to access the database. - `destinationType` 'mssql-v2', required - DestinationMysql - `database` string, required — Name of the database. - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `host` string, required — Hostname of the database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `password` string — Password associated with the username. - `port` integer, required — Port of the database. - `raw_data_schema` string — The database to write raw tables into - `ssl` boolean — Encrypt data using SSL. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `username` string, required — Username to use to access the database. - `destinationType` 'mysql', required - DestinationOracle - `encryption` union — The encryption method which is used when communicating with the database. - object — Data transfer will not be encrypted. - `encryption_method` 'unencrypted', required - object — The native network encryption gives you the ability to encrypt database connections, without the configuration overhead of TCP/IP and SSL/TLS and without the need to open and listen on different ports. - `encryption_algorithm` 'AES256' | 'RC4_56' | '3DES168' — This parameter defines the database encryption algorithm. - `encryption_method` 'client_nne', required - object — Verify and use the certificate provided by the server. - `encryption_method` 'encrypted_verify_certificate', required - `ssl_certificate` string, required — Privacy Enhanced Mail (PEM) files are concatenated certificate containers frequently used in certificate installations. - `host` string, required — The hostname of the database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `password` string — The password associated with the username. - `port` integer, required — The port of the database. - `raw_data_schema` string — The schema to write raw tables into (default: airbyte_internal) - `schema` string — The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. The usual value for this field is "airbyte". In Oracle, schemas and users are the same thing, so the "user" parameter is used as the login credentials and this is used for the default Airbyte message schema. - `sid` string, required — The System Identifier uniquely distinguishes the instance from any other instance on the same computer. - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `username` string, required — The username to access the database. This user must have CREATE USER privileges in the database. - `destinationType` 'oracle', required - DestinationPgvector — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Postgres can be used to store vector data and retrieve embeddings. - `credentials` object, required - `password` string, required — Enter the password you want to use to access the database - `database` string, required — Enter the name of the database that you want to sync data into - `default_schema` string — Enter the name of the default schema - `host` string, required — Enter the account name you want to use to access the database. - `port` integer — Enter the port you want to use to access the database - `username` string, required — Enter the name of the user you want to use to access the database - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'pgvector', required - DestinationPinecone — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Pinecone is a popular vector store that can be used to store and retrieve embeddings. - `index` string, required — Pinecone index in your project to load data into - `pinecone_environment` string, required — Pinecone Cloud environment to use - `pinecone_key` string, required — The Pinecone API key to use matching the environment (copy from Pinecone console) - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'pinecone', required - DestinationPostgres - `database` string, required — Name of the database. - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `drop_cascade` boolean — Drop tables with CASCADE. WARNING! This will delete all data in all dependent objects (views, etc.). Use with caution. This option is intended for usecases which can easily rebuild the dependent objects. - `host` string, required — Hostname of the database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `password` string — Password associated with the username. - `port` integer, required — Port of the database. - `raw_data_schema` string — The schema to write raw tables into - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl` boolean — Encrypt data using SSL. When activating SSL, please select one of the connection modes. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the source database prefer - Chose this mode to allow unencrypted connection only if the source database does not support encryption require - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `ca_certificate` string, required — CA certificate - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `mode` 'verify-ca', required - object — Verify-full SSL mode. - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `mode` 'verify-full', required - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `unconstrained_number` boolean — Create numeric columns as unconstrained DECIMAL instead of NUMBER(38, 9). This will allow increased precision in numeric values. (this is disabled by default for backwards compatibility, but is recommended to enable) - `username` string, required — Username to use to access the database. - `destinationType` 'postgres', required - DestinationPubsub - `batching_delay_threshold` integer — Number of ms before the buffer is flushed - `batching_element_count_threshold` integer — Number of messages before the buffer is flushed - `batching_enabled` boolean, required — If TRUE messages will be buffered instead of sending them one by one - `batching_request_bytes_threshold` integer — Number of bytes before the buffer is flushed - `credentials_json` string, required — The contents of the JSON service account key. Check out the docs if you need help generating this key. - `ordering_enabled` boolean, required — If TRUE PubSub publisher will have message ordering enabled. Every message will have an ordering key of stream - `project_id` string, required — The GCP project ID for the project containing the target PubSub. - `topic_id` string, required — The PubSub topic ID in the given GCP project ID. - `destinationType` 'pubsub', required - DestinationQdrant — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Indexing configuration - `auth_method` union — Method to authenticate with the Qdrant Instance - object - … - object - … - `collection` string, required — The collection to load data into - `distance_metric` 'dot' | 'cos' | 'euc' — The Distance metric used to measure similarities among vectors. This field is only used if the collection defined in the does not exist yet and is created automatically by the connector. - `prefer_grpc` boolean — Whether to prefer gRPC over HTTP. Set to true for Qdrant cloud clusters - `text_field` string — The field in the payload that contains the embedded text - `url` string, required — Public Endpoint of the Qdrant cluser - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'qdrant', required - DestinationRedis - `cache_type` 'hash', required — Redis cache type to store data in. - `host` string, required — Redis host to connect to. - `password` string — Password associated with Redis. - `port` integer, required — Port of Redis. - `ssl` boolean — Indicates whether SSL encryption protocol will be used to connect to Redis. It is recommended to use SSL connection if possible. - `ssl_mode` union — SSL connection modes.
  • verify-full - This is the most secure mode. Always require encryption and verifies the identity of the source database server - object — Disable SSL. - `mode` 'disable', required - object — Verify-full SSL mode. - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. If you do not add it - the password will be generated automatically. - `mode` 'verify-full', required - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `username` string, required — Username associated with Redis. - `destinationType` 'redis', required - DestinationRedshift - `database` string, required — Name of the database. - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `drop_cascade` boolean — Drop tables with CASCADE. WARNING! This will delete all data in all dependent objects (views, etc.). Use with caution. This option is intended for usecases which can easily rebuild the dependent objects. - `host` string, required — Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com) - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `password` string, required — Password associated with the username. - `port` integer, required — Port of the database. - `raw_data_schema` string — The schema to write raw tables into (default: airbyte_internal). - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is "public". - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `uploading_method` object — (recommended) Uploads data to S3 and then uses a COPY to insert the data into Redshift. COPY is recommended for production workloads for better speed and scalability. See AWS docs for more details. - `access_key_id` string, required — This ID grants access to the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. See AWS docs on how to generate an access key ID and secret access key. - `file_name_pattern` string — The pattern allows you to set the file-name format for the S3 staging file(s) - `method` 'S3 Staging', required - `purge_staging_data` boolean — Whether to delete the staging files from S3 after completing the sync. See docs for details. - `s3_bucket_name` string, required — The name of the staging S3 bucket. - `s3_bucket_path` string — The directory under the S3 bucket where data will be written. If not provided, then defaults to the root directory. See path's name recommendations for more details. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 staging bucket. - `secret_access_key` string, required — The corresponding secret to the above access key id. See AWS docs on how to generate an access key ID and secret access key. - `username` string, required — Username to use to access the database. - `destinationType` 'redshift', required - DestinationS3 - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `file_name_pattern` string — Pattern to match file names in the bucket directory. Read more here - `format` union, required — Format of the data output. See here for more details - object - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - … - `flattening` 'No flattening' | 'Root level flattening', required - `format_type` 'CSV', required - object - `compression` union — Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). - … - `flattening` 'No flattening' | 'Root level flattening' - `format_type` 'JSONL', required - object - `compression_codec` union, required — The compression algorithm used to compress data. Default to no compression. - … - `format_type` 'Avro', required - object - `block_size_mb` integer — This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB. - `compression_codec` 'UNCOMPRESSED' | 'SNAPPY' | 'GZIP' | 'LZO' | 'BROTLI' | 'LZ4' | 'ZSTD' — The compression algorithm used to compress data pages. - `dictionary_encoding` boolean — Default: true. - `dictionary_page_size_kb` integer — There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB. - `format_type` 'Parquet', required - `max_padding_size_mb` integer — Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB. - `page_size_kb` integer — The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB. - `role_arn` string — The ARN of the AWS role to assume. Only usable in Airbyte Cloud. - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_path` string, required — Directory under the S3 bucket where data will be written. Read more here - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `s3_path_format` string — Format string on how data will be organized inside the bucket directory. Read more here - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `destinationType` 's3', required - DestinationS3DataLake — Defines the configurations required to connect to an Iceberg catalog, including warehouse location, main branch name, and catalog type specifics. - `access_key_id` string — The AWS Access Key ID with permissions for S3 and Glue operations. - `catalog_type` union, required — Specifies the type of Iceberg catalog (e.g., NESSIE, GLUE, REST, POLARIS) and its associated configuration. - object — Configuration details for connecting to a Nessie-based Iceberg catalog. - `access_token` string — Optional token for authentication with the Nessie server. - `catalog_type` 'NESSIE', required - `namespace` string, required — The Nessie namespace to be used in the Table identifier. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined` - `server_uri` string, required — The base URL of the Nessie server used to connect to the Nessie catalog. - object — Configuration details for connecting to an AWS Glue-based Iceberg catalog. - `catalog_type` 'GLUE', required - `database_name` string, required — The Glue database name. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined` - `glue_id` string, required — The AWS Account ID associated with the Glue service used by the Iceberg catalog. - `role_arn` string — The ARN of the AWS role to assume. Only usable in Airbyte Cloud. - object — Configuration details for connecting to a REST catalog. - `catalog_type` 'REST', required - `namespace` string, required — The namespace to be used in the Table identifier. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined` - `server_uri` string, required — The base URL of the Rest server used to connect to the Rest catalog. - object — Configuration details for connecting to an Apache Polaris-based Iceberg catalog. - `catalog_name` string, required — The name of the catalog in Polaris. This corresponds to the catalog name created via the Polaris Management API. - `catalog_type` 'POLARIS', required - `client_id` string, required — The OAuth Client ID for authenticating with the Polaris server. - `client_secret` string, required — The OAuth Client Secret for authenticating with the Polaris server. - `namespace` string, required — The Polaris namespace to be used in the Table identifier. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined` - `server_uri` string, required — The base URL of the Polaris server used to connect to the Polaris catalog. - `main_branch_name` string, required — The primary or default branch name in the catalog. Most query engines will use "main" by default. See Iceberg documentation for more information. - `s3_bucket_name` string, required — The name of the S3 bucket that will host the Iceberg data. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `secret_access_key` string — The AWS Secret Access Key paired with the Access Key ID for AWS authentication. - `warehouse_location` string, required — The root location of the data warehouse used by the Iceberg catalog. Typically includes a bucket name and path within that bucket. For AWS Glue and Nessie, must include the storage protocol (such as "s3://" for Amazon S3). - `destinationType` 's3-data-lake', required - DestinationSalesforce - `auth_type` 'Client', required - `client_id` string, required — Enter your Salesforce developer application's Client ID. - `client_secret` string, required — Enter your Salesforce developer application's Client secret. - `is_sandbox` boolean, required — Toggle if you're using a Salesforce Sandbox. - `object_storage_config` union - object - `storage_type` 'None', required - object - `access_key_id` string — The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here. - `bucket_path` string, required — All files in the bucket will be prefixed by this. - `role_arn` string — The ARN of the AWS role to assume. Only usable in Airbyte Cloud. - `s3_bucket_name` string, required — The name of the S3 bucket. Read more here. - `s3_bucket_region` '' | 'af-south-1' | 'ap-east-1' | 'ap-northeast-1' | 'ap-northeast-2' | 'ap-northeast-3' | 'ap-south-1' | 'ap-south-2' | 'ap-southeast-1' | 'ap-southeast-2' | 'ap-southeast-3' | 'ap-southeast-4' | 'ca-central-1' | 'ca-west-1' | 'cn-north-1' | 'cn-northwest-1' | 'eu-central-1' | 'eu-central-2' | 'eu-north-1' | 'eu-south-1' | 'eu-south-2' | 'eu-west-1' | 'eu-west-2' | 'eu-west-3' | 'il-central-1' | 'me-central-1' | 'me-south-1' | 'sa-east-1' | 'us-east-1' | 'us-east-2' | 'us-gov-east-1' | 'us-gov-west-1' | 'us-west-1' | 'us-west-2', required — The region of the S3 bucket. See here for all region codes. - `s3_endpoint` string — Your S3 endpoint url. Read more here - `secret_access_key` string — The corresponding secret to the access key ID. Read more here - `storage_type` 'S3', required - `refresh_token` string, required — Enter your application's Salesforce Refresh Token used for Airbyte to access your Salesforce account. - `destinationType` 'salesforce', required - DestinationSftpJson - `destination_path` string, required — Path to the directory where json files will be written. - `host` string, required — Hostname of the SFTP server. - `password` string, required — Password associated with the username. - `port` integer — Port of the SFTP server. - `username` string, required — Username to use to access the SFTP server. - `destinationType` 'sftp-json', required - DestinationSnowflake - `cdc_deletion_mode` 'Hard delete' | 'Soft delete' — Whether to execute CDC deletions as hard deletes (i.e. propagate source deletions to the destination), or soft deletes (i.e. leave a tombstone record in the destination). Defaults to hard deletes. - `credentials` union — Determines the type of authentication that should be used. - object — Configuration details for the Key Pair Authentication. - `auth_type` 'Key Pair Authentication', required - `private_key` string, required — RSA Private key to use for Snowflake connection. See the docs for more information on how to obtain this key. - `private_key_password` string — Passphrase for private key - object — Configuration details for the Username and Password Authentication. - `auth_type` 'Username and Password', required - `password` string, required — Enter the password associated with the username. - `database` string, required — Enter the name of the database you want to sync data into - `disable_type_dedupe` boolean — Write the legacy "raw tables" format, to enable backwards compatibility with older versions of this connector. - `host` string, required — Enter your Snowflake account's locator (in the format ...snowflakecomputing.com) - `jdbc_url_params` string — Enter the additional properties to pass to the JDBC URL string when connecting to the database (formatted as key=value pairs separated by the symbol &). Example: key1=value1&key2=value2&key3=value3 - `raw_data_schema` string — Airbyte will use this dataset for various internal tables. In legacy raw tables mode, the raw tables will be stored in this dataset. Defaults to "airbyte_internal". - `retention_period_days` integer — The number of days of Snowflake Time Travel to enable on the tables. See Snowflake's documentation for more information. Setting a nonzero value will incur increased storage costs in your Snowflake instance. - `role` string, required — Enter the role that you want to use to access Snowflake - `schema` string, required — Enter the name of the default schema - `username` string, required — Enter the name of the user you want to use to access the database - `warehouse` string, required — Enter the name of the warehouse that you want to use as a compute cluster - `destinationType` 'snowflake', required - DestinationSnowflakeCortex — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Snowflake can be used to store vector data and retrieve embeddings. - `credentials` object, required - `password` string, required — Enter the password you want to use to access the database - `database` string, required — Enter the name of the database that you want to sync data into - `default_schema` string, required — Enter the name of the default schema - `host` string, required — Enter the account name you want to use to access the database. This is usually the identifier before .snowflakecomputing.com - `role` string, required — Enter the role that you want to use to access Snowflake - `username` string, required — Enter the name of the user you want to use to access the database - `warehouse` string, required — Enter the name of the warehouse that you want to use as a compute cluster - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'snowflake-cortex', required - DestinationSurrealdb - `surrealdb_database` string, required — The database to use in SurrealDB. - `surrealdb_namespace` string, required — The namespace to use in SurrealDB. - `surrealdb_password` string, required — The password to use in SurrealDB. - `surrealdb_url` string, required — The URL of the SurrealDB instance. - `surrealdb_username` string, required — The username to use in SurrealDB. - `destinationType` 'surrealdb', required - DestinationTeradata - `disable_type_dedupe` boolean — Disable Writing Final Tables. WARNING! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions - `drop_cascade` boolean — Drop tables with CASCADE. WARNING! This will delete all data in all dependent objects (views, etc.). Use with caution. This option is intended for usecases which can easily rebuild the dependent objects. - `host` string, required — Hostname of the database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `logmech` union - object - `auth_type` 'TD2' - `password` string, required — Enter the password associated with the username. - `username` string, required — Username to use to access the database. - object - `auth_type` 'LDAP' - `password` string, required — Enter the password associated with the username. - `username` string, required — Username to use to access the database. - `query_band` string — Defines the custom session query band using name-value pairs. For example, 'org=Finance;report=Fin123;' - `raw_data_schema` string — The database to write raw tables into - `schema` string — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl` boolean — Encrypt data using SSL. When activating SSL, please select one of the SSL modes. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the destination database prefer - Chose this mode to allow unencrypted connection only if the destination database does not support encryption require - Chose this mode to always require encryption. If the destination database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the destination database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the destination database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `mode` 'verify-ca', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-ca. See more information - in the docs. - object — Verify-full SSL mode. - `mode` 'verify-full', required - `ssl_ca_certificate` string, required — Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with SSLMODE=verify-full. See more information - in the docs. - `destinationType` 'teradata', required - DestinationTimeplus - `apikey` string, required — Personal API key - `endpoint` string, required — Timeplus workspace endpoint - `destinationType` 'timeplus', required - DestinationTypesense - `api_key` string, required — Typesense API Key - `batch_size` integer — How many documents should be imported together. Default 1000 - `host` string, required — Hostname of the Typesense instance without protocol. Accept multiple hosts separated by comma. - `path` string — Path of the Typesense instance. Default is none - `port` string — Port of the Typesense instance. Ex: 8108, 80, 443. Default is 8108 - `protocol` string — Protocol of the Typesense instance. Ex: http or https. Default is https - `destinationType` 'typesense', required - DestinationVectara — Configuration to connect to the Vectara instance - `corpus_name` string, required — The Name of Corpus to load data into - `customer_id` string, required — Your customer id as it is in the authenticaion url - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `oauth2` object, required — OAuth2.0 credentials used to authenticate admin actions (creating/deleting corpora) - `client_id` string, required — OAuth2.0 client id - `client_secret` string, required — OAuth2.0 client secret - `parallelize` boolean — Parallelize indexing into Vectara with multiple threads - `text_fields` string[] — List of fields in the record that should be in the section of the document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `title_field` string — A field that will be used to populate the `title` of each document. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `destinationType` 'vectara', required - DestinationWeaviate — The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination. The configuration model is composed of four parts: * Processing configuration * Embedding configuration * Indexing configuration * Advanced configuration Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. - `embedding` union, required — Embedding configuration - object — Do not calculate and pass embeddings to Weaviate. Suitable for clusters with configured vectorizers to calculate embeddings within Weaviate or for classes that should only support regular text search. - `mode` 'no_embedding', required - object — Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `api_base` string, required — The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `deployment` string, required — The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - `mode` 'azure_openai', required - `openai_key` string, required — The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource - object — Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. - `mode` 'openai', required - `openai_key` string, required - object — Use the Cohere API to embed text. - `cohere_key` string, required - `mode` 'cohere', required - object — Use a field in the record as the embedding. This is useful if you already have an embedding for your data and want to store it in the vector store. - `dimensions` integer, required — The number of dimensions the embedding model is generating - `field_name` string, required — Name of the field in the record that contains the embedding - `mode` 'from_field', required - object — Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. - `mode` 'fake', required - object — Use a service that's compatible with the OpenAI API to embed text. - `api_key` string - `base_url` string, required — The base URL for your OpenAI-compatible service - `dimensions` integer, required — The number of dimensions the embedding model is generating - `mode` 'openai_compatible', required - `model_name` string — The name of the model to use for embedding - `indexing` object, required — Indexing configuration - `additional_headers` object[] — Additional HTTP headers to send with every request. - `header_key` string, required - `value` string, required - `auth` union, required — Authentication method - object — Authenticate using an API token (suitable for Weaviate Cloud) - … - object — Authenticate using username and password (suitable for self-managed Weaviate clusters) - … - object — Do not authenticate (suitable for locally running test clusters, do not use for clusters with public IP addresses) - … - `batch_size` integer — The number of records to send to Weaviate in each batch - `default_vectorizer` 'none' | 'text2vec-cohere' | 'text2vec-huggingface' | 'text2vec-openai' | 'text2vec-palm' | 'text2vec-contextionary' | 'text2vec-transformers' | 'text2vec-gpt4all' — The vectorizer to use if new classes need to be created - `host` string, required — The public endpoint of the Weaviate cluster. - `tenant_id` string — The tenant ID to use for multi tenancy - `text_field` string — The field in the object that contains the embedded text - `omit_raw_text` boolean — Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. - `processing` object, required - `chunk_overlap` integer — Size of overlap between chunks in tokens to store in vector store to better capture relevant context - `chunk_size` integer, required — Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) - `field_name_mappings` object[] — List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. - `from_field` string, required — The field name in the source - `to_field` string, required — The field name to use in the destination - `metadata_fields` string[] — List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. - `text_fields` string[] — List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. `user.name` will access the `name` field in the `user` object. It's also possible to use wildcards to access all fields in an object, e.g. `users.*.name` will access all `names` fields in all entries of the `users` array. - `text_splitter` union — Split text fields into chunks based on the specified method. - object — Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. - … - object — Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. - … - object — Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. - … - `destinationType` 'weaviate', required - DestinationYellowbrick - `database` string, required — Name of the database. - `host` string, required — Hostname of the database. - `jdbc_url_params` string — Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3). - `password` string — Password associated with the username. - `port` integer, required — Port of the database. - `schema` string, required — The default schema tables are written to if the source does not specify a namespace. The usual value for this field is "public". - `ssl` boolean — Encrypt data using SSL. When activating SSL, please select one of the connection modes. - `ssl_mode` union — SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the source database prefer - Chose this mode to allow unencrypted connection only if the source database does not support encryption require - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server See more information - in the docs. - object — Disable SSL. - `mode` 'disable', required - object — Allow SSL mode. - `mode` 'allow', required - object — Prefer SSL mode. - `mode` 'prefer', required - object — Require SSL mode. - `mode` 'require', required - object — Verify-ca SSL mode. - `ca_certificate` string, required — CA certificate - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `mode` 'verify-ca', required - object — Verify-full SSL mode. - `ca_certificate` string, required — CA certificate - `client_certificate` string, required — Client certificate - `client_key` string, required — Client key - `client_key_password` string — Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically. - `mode` 'verify-full', required - `tunnel_method` union — Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use. - object - `tunnel_method` 'NO_TUNNEL', required — No ssh tunnel needed to connect to database - object - `ssh_key` string, required — OS-level user account ssh key credentials in RSA PEM format ( created with ssh-keygen -t rsa -m PEM -f myuser_rsa ) - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_KEY_AUTH', required — Connect through a jump server tunnel host using username and ssh key - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host. - object - `tunnel_host` string, required — Hostname of the jump server host that allows inbound ssh tunnel. - `tunnel_method` 'SSH_PASSWORD_AUTH', required — Connect through a jump server tunnel host using username and password authentication - `tunnel_port` integer, required — Port on the proxy/jump server that accepts inbound ssh connections. - `tunnel_user` string, required — OS-level username for logging into the jump server host - `tunnel_user_password` string, required — OS-level password for logging into the jump server host - `username` string, required — Username to use to access the database. - `destinationType` 'yellowbrick', required - `createdAt` integer, required - `resourceAllocation` ScopedResourceRequirements — actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level. - `default` ResourceRequirements — optional resource requirements to run workers (blank for unbounded allocations) - `cpu_request` string - `cpu_limit` string - `memory_request` string - `memory_limit` string - `ephemeral_storage_request` string - `ephemeral_storage_limit` string - `jobSpecific` JobTypeResourceLimit[] - `jobType` 'get_spec' | 'check_connection' | 'discover_schema' | 'sync' | 'reset_connection' | 'connection_updater' | 'replicate', required — enum that describes the different types of jobs that the platform runs. - `resourceRequirements` ResourceRequirements, required — optional resource requirements to run workers (blank for unbounded allocations) - `cpu_request` string - `cpu_limit` string - `memory_request` string - `memory_limit` string - `ephemeral_storage_request` string - `ephemeral_storage_limit` string ## Other responses - `403` — Not allowed - `404` — Not found --- [API](https://skmtc.net/airbytehq/apis/airbyte-api.md) · [All operations](https://skmtc.net/airbytehq/apis/airbyte-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/airbytehq/airbyte-api/revisions/55ddc054e5b5/schema)