v1

latestOpenAPI 3.0.12026-07-17111315.3 KB

Submission

Submit a Spark application

To submit a job, prepare a job payload in the request body either in JSON or YAML format.

post/spark

content-typestring

options: application/json, text/yaml, or leave it empty for API to figure out

submissionIdSuffixstring

If specified, the suffix will be appended to the submission ID

applicationNamestring

Name of the application. This will help listing submissions under the same application later.

typestring

Should be: Java / Scala / Python. If not specified, the API will try to figure out by itself.

imagestring

spotInstanceboolean

To enable Spot Instance feature that schedules all Spark Executor pods to Spot nodes

sparkVersionstring required

Spark version in the format of x.y, where x and y are integers.

mainClassstring

The main class in the jar provided in mainApplicationFile for a Java/Scala Spark job

mainApplicationFilestring required

For Java/Scala Spark jobs, provide the full path to the jar file. For PySpark jobs, provide the full path to the Python file.

argumentsstring[]

annotationsobject

pythonVersionstring

queuestring

If no queue is specified, a default 'poc' queue will be used.

{
  "mainClass": "com.spark.examples.OneStageApp"
}

submissionIdstring