Skip to content

Spark on Kubernetes

Submit and observe Spark applications from Python, the CLI or Airflow using your existing Kubernetes cluster. Optional API and shared web-console components make applications, logs and diagnostics accessible to operators. Spark creates the executors; your cluster and applications retain their infrastructure and data responsibilities.

What's new

The release notes list every published release and the release in preparation, whichever documentation version you are reading.

Start here Install

You can use the core Python client or CLI without deploying the API or Airflow. Spark and Java run in the application image; they are not required on the submitting machine. See Spark compatibility for image requirements.

Choose what to run

  • Submit an application

    Run an existing Python or JVM application from the Python client or the CLI, then wait for it, stream its logs or cancel it.

    Your first Kubernetes job

  • Portable job files and SQL

    Describe jobs in versioned files with profiles, preview them offline, and run SQL files or inline statements.

    Job files · SQL jobs

  • Spark Connect and notebooks

    Use an existing Spark Connect endpoint, or create a private server for a notebook or interactive session.

    Spark Connect · Jupyter

  • Schedule with Airflow

    Run Spark applications and SQL from Airflow 2.11 or 3, with deferrable waiting and a native Airflow 3 task tab.

    Airflow

  • Observe and diagnose

    List applications, read logs and open the Spark UI from the web console, and explain Pending or failed drivers.

    Diagnostics · Run history

  • Deploy the API

    Install the API, console and optional History Server with the Helm chart, and accept approved remote submissions.

    Chart deployment · Remote submission

Try the ETL starter locally before configuring a cluster, or run dbt with temporary Spark compute through the experimental dbt adapter.

How the pieces fit

flowchart LR
  subgraph clients [Where you submit]
    sdk[Python client]
    cli[CLI]
    airflow[Airflow operators]
    remote[Remote CLI]
  end
  api[API and web console]
  subgraph cluster [Your Kubernetes namespace]
    driver[Spark driver pod]
    executors[Executor pods]
    history[Spark History Server]
  end
  sdk & cli & airflow --> k8s[(Kubernetes API)]
  remote -- approved jobs --> api
  api --> k8s
  k8s --> driver
  driver -- creates --> executors
  api -. logs, Spark UI proxy .-> driver
  api -. proxy .-> history

Every route creates the driver pod through the Kubernetes API, and Spark then creates and owns the executors. Local routes use your Kubernetes credentials; remote submission uses the API's service account after its policy admits the job. The API is optional: it adds a browser console, proxies and policy-gated remote submission on top of the same client.

Before you rely on it

The develop documentation follows main. Before copying its examples, choose a matching published release or a source installation. Installing the PyPI package does not include the repository's example files. Chart and package releases are independent. Read support boundaries before planning production operation: this project does not provide a multi-tenant scheduler, a highly available metadata service or an embedded identity provider. When to use it compares it with Spark operators, Airflow pod tasks and managed Spark services.

Contributors can use the repository map and check routes. Maintainers use the release runbook to qualify and publish artifacts.