--- title: "POST /v1/{+parent}/batches" method: POST path: "/v1/{+parent}/batches" tags: ["projects"] --- # POST /v1/{+parent}/batches `POST /v1/{+parent}/batches` Creates a batch workload that executes asynchronously. ## Path parameters - `parent` string, required ## Query parameters - `batchId` string - `requestId` string ## Request body - Batch — A representation of a batch workload in the service. - `state` 'STATE_UNSPECIFIED' | 'PENDING' | 'RUNNING' | 'CANCELLING' | 'CANCELLED' | 'SUCCEEDED' | 'FAILED' — Output only. The state of the batch. - `runtimeConfig` RuntimeConfig — Runtime configuration for a workload. - `properties` object — Optional. A mapping of property names to values, which are used to configure workload execution. - `cohort` string — Optional. Cohort identifier. Identifies families of the workloads that have the same shape, for example, daily ETL jobs. - `autotuningConfig` AutotuningConfig — Autotuning configuration of the workload. - `scenarios` string[] — Optional. Scenarios for which tunings are applied. - `version` string — Optional. Version of the batch runtime. - `containerImage` string — Optional. Optional custom container image for the job runtime environment. If not specified, a default container image will be used. - `repositoryConfig` RepositoryConfig — Configuration for dependency repositories - `pypiRepositoryConfig` PyPiRepositoryConfig — Configuration for PyPi repository - `pypiRepository` string — Optional. The PyPi repository address. Note: This field is not available for batch workloads. - `createTime` string, google-datetime — Output only. The time when the batch was created. - `runtimeInfo` RuntimeInfo — Runtime information about workload execution. - `endpoints` object — Output only. Map of remote access endpoints (such as web interfaces and APIs) to their URIs. - `outputUri` string — Output only. A URI pointing to the location of the stdout and stderr of the workload. - `propertiesInfo` PropertiesInfo — Properties of the workload organized by origin. - `autotuningProperties` object — Output only. Properties set by autotuning engine. - `diagnosticOutputUri` string — Output only. A URI pointing to the location of the diagnostics tarball. - `approximateUsage` UsageMetrics — Usage metrics represent approximate total resources consumed by a workload. - `shuffleStorageGbSeconds` string, int64 — Optional. Shuffle storage usage in (GB x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `updateTime` string, google-datetime — Optional. The timestamp of the usage metrics. - `milliAcceleratorSecondsL4` string, int64 — Optional. L4 accelerator usage in (milliAccelerator x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `acceleratorType` string — Optional. Accelerator type being used, if any Deprecated: This field is only used in runtime versions below 3.0. - `milliDcuSeconds` string, int64 — Optional. DCU (Dataproc Compute Units) usage in (milliDCU x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `milliAcceleratorSecondsA10040` string, int64 — Optional. A100-40 accelerator usage in (milliAccelerator x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `milliAcceleratorSecondsA10080` string, int64 — Optional. A100-80 accelerator usage in (milliAccelerator x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `milliAcceleratorSeconds` string, int64 — Optional. Accelerator usage in (milliAccelerator x seconds) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). Deprecated: This field is only used in runtime versions below 3.0. - `currentUsage` UsageSnapshot — The usage snapshot represents the resources consumed by a workload at a specified time. - `milliAcceleratorA10080` string, int64 — Optional. Milli (one-thousandth) accelerator for A100-80 accelerators. (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) - `shuffleStorageGbPremium` string, int64 — Optional. Shuffle Storage in gigabytes (GB) charged at premium tier. (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) - `shuffleStorageGb` string, int64 — Optional. Shuffle Storage in gigabytes (GB). (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) - `milliDcuPremium` string, int64 — Optional. Milli (one-thousandth) Dataproc Compute Units (DCUs) charged at premium tier (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `milliAccelerator` string, int64 — Optional. Milli (one-thousandth) accelerator. (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) Deprecated: This field is only used in runtime versions below 3.0. - `milliDcu` string, int64 — Optional. Milli (one-thousandth) Dataproc Compute Units (DCUs) (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)). - `acceleratorType` string — Optional. Accelerator type being used, if any Deprecated: This field is only used in runtime versions below 3.0. - `milliAcceleratorA10040` string, int64 — Optional. Milli (one-thousandth) accelerator for A100-40 accelerators. (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) - `snapshotTime` string, google-datetime — Optional. The timestamp of the usage snapshot. - `milliAcceleratorL4` string, int64 — Optional. Milli (one-thousandth) accelerator for L4 accelerators. (see Dataproc Serverless pricing (https://cloud.google.com/dataproc-serverless/pricing)) - `cohortInfo` CohortInfo — Information about the cohort that the workload belongs to. - `cohort` string — Output only. Final cohort that was used to tune the workload. - `cohortSource` 'COHORT_SOURCE_UNSPECIFIED' | 'USER_PROVIDED' | 'AIRFLOW' — Output only. Source of the cohort. - `environmentConfig` EnvironmentConfig — Environment configuration for a workload. - `executionConfig` ExecutionConfig — Execution configuration for a workload. - `authenticationConfig` AuthenticationConfig — Authentication configuration for a workload is used to set the default identity for the workload execution. The config specifies the type of identity (service account or user) that will be used by workloads to access resources on the project(s). - `userWorkloadAuthenticationType` 'AUTHENTICATION_TYPE_UNSPECIFIED' | 'SERVICE_ACCOUNT' | 'END_USER_CREDENTIALS' — Optional. Authentication type for the user workload running in containers. - `resourceManagerTags` object — Optional. Associates Resource Manager tags with the workload nodes. There is a max limit of 30 tags. Keys and values can be either in numeric format, such as tagKeys/{tag_key_id} and tagValues/{tag_value_id}, or in namespaced format, such as {org_id|project_id}/{tag_key_short_name} and {tag_value_short_name}. - `subnetworkUri` string — Optional. Subnetwork URI to connect workload to. - `stagingBucket` string — Optional. A Cloud Storage bucket used to stage workload dependencies, config files, and store workload output and other ephemeral data, such as Spark history files. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location according to the region where your workload is running, and then create and manage project-level, per-location staging and temporary buckets. This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket. - `networkTags` string[] — Optional. Tags used for network traffic control. - `networkUri` string — Optional. Network URI to connect workload to. - `kmsKey` string — Optional. The Cloud KMS key to use for encryption. - `serviceAccount` string — Optional. Service account that used to execute workload. - `ttl` string, google-duration — Optional. The duration after which the workload will be terminated, specified as the JSON representation for Duration (https://protobuf.dev/programming-guides/proto3/#json). When the workload exceeds this duration, it will be unconditionally terminated without waiting for ongoing work to finish. If ttl is not specified for a batch workload, the workload will be allowed to run until it exits naturally (or run forever without exiting). If ttl is not specified for an interactive session, it defaults to 24 hours. If ttl is not specified for a batch that uses 2.1+ runtime version, it defaults to 4 hours. Minimum value is 10 minutes; maximum value is 14 days. If both ttl and idle_ttl are specified (for an interactive session), the conditions are treated as OR conditions: the workload will be terminated when it has been idle for idle_ttl or when ttl has been exceeded, whichever occurs first. - `idleTtl` string, google-duration — Optional. Applies to sessions only. The duration to keep the session alive while it's idling. Exceeding this threshold causes the session to terminate. This field cannot be set on a batch workload. Minimum value is 10 minutes; maximum value is 14 days (see JSON representation of Duration (https://developers.google.com/protocol-buffers/docs/proto3#json)). Defaults to 1 hour if not set. If both ttl and idle_ttl are specified for an interactive session, the conditions are treated as OR conditions: the workload will be terminated when it has been idle for idle_ttl or when ttl has been exceeded, whichever occurs first. - `peripheralsConfig` PeripheralsConfig — Auxiliary services configuration for a workload. - `metastoreService` string — Optional. Resource name of an existing Dataproc Metastore service.Example: projects/[project_id]/locations/[region]/services/[service_id] - `sparkHistoryServerConfig` SparkHistoryServerConfig — Spark History Server configuration for the workload. - `dataprocCluster` string — Optional. Resource name of an existing Dataproc Cluster to act as a Spark History Server for the workload.Example: projects/[project_id]/regions/[region]/clusters/[cluster_name] - `sparkRBatch` SparkRBatch — A configuration for running an Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) batch workload. - `mainRFileUri` string — Required. The HCFS URI of the main R file to use as the driver. Must be a .R or .r file. - `args` string[] — Optional. The arguments to pass to the Spark driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission. - `archiveUris` string[] — Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. - `fileUris` string[] — Optional. HCFS URIs of files to be placed in the working directory of each executor. - `name` string — Output only. The resource name of the batch. - `pysparkBatch` PySparkBatch — A configuration for running an Apache PySpark (https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html) batch workload. - `jarFileUris` string[] — Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks. - `pythonFileUris` string[] — Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip. - `archiveUris` string[] — Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. - `fileUris` string[] — Optional. HCFS URIs of files to be placed in the working directory of each executor. - `args` string[] — Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission. - `mainPythonFileUri` string — Required. The HCFS URI of the main Python file to use as the Spark driver. Must be a .py file. - `uuid` string — Output only. A batch UUID (Unique Universal Identifier). The service generates this value when it creates the batch. - `sparkSqlBatch` SparkSqlBatch — A configuration for running Apache Spark SQL (https://spark.apache.org/sql/) queries as a batch workload. - `queryVariables` object — Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";). - `jarFileUris` string[] — Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH. - `queryFileUri` string — Required. The HCFS URI of the script that contains Spark SQL queries to execute. - `sparkBatch` SparkBatch — A configuration for running an Apache Spark (https://spark.apache.org/) batch workload. - `mainJarFileUri` string — Optional. The HCFS URI of the jar file that contains the main class. - `jarFileUris` string[] — Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks. - `archiveUris` string[] — Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. - `fileUris` string[] — Optional. HCFS URIs of files to be placed in the working directory of each executor. - `mainClass` string — Optional. The name of the driver main class. The jar file that contains the class must be in the classpath or specified in jar_file_uris. - `args` string[] — Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission. - `labels` object — Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch. - `stateHistory` StateHistory[] — Output only. Historical state information for the batch. - `stateMessage` string — Output only. Details about the state at this point in history. - `stateStartTime` string, google-datetime — Output only. The time when the batch entered the historical state. - `state` 'STATE_UNSPECIFIED' | 'PENDING' | 'RUNNING' | 'CANCELLING' | 'CANCELLED' | 'SUCCEEDED' | 'FAILED' — Output only. The state of the batch at this point in history. - `operation` string — Output only. The resource name of the operation associated with this batch. - `stateMessage` string — Output only. Batch state details, such as a failure description if the state is FAILED. - `stateTime` string, google-datetime — Output only. The time when the batch entered a current state. - `creator` string — Output only. The email address of the user who created the batch. - `pysparkNotebookBatch` PySparkNotebookBatch — A configuration for running a PySpark Notebook batch workload. - `pythonFileUris` string[] — Optional. HCFS URIs of Python files to pass to the PySpark framework. - `archiveUris` string[] — Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. - `jarFileUris` string[] — Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH. - `notebookFileUri` string — Required. The HCFS URI of the notebook file to execute. - `params` object — Optional. The parameters to pass to the notebook. - `fileUris` string[] — Optional. HCFS URIs of files to be placed in the working directory of each executor ## Response `200` Successful response --- [API](https://skmtc.net/google/apis/dataproc.md) · [All operations](https://skmtc.net/google/apis/dataproc/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/google/dataproc/versions/15847ccbcf7d/schema)