Python reference¶
Start with the documented imports below. Each reference page shows signatures,
parameters and return types from this documentation revision. Choose the docs
version that matches your installed package; develop can describe unreleased
changes.
Client and results¶
| Task | Entry point or result | Usage guide |
|---|---|---|
| Submit and observe Spark applications | SparkOnK8S from spark_on_k8s.client |
Getting started |
| Set driver and executor resources | PodResources and ExecutorInstances from spark_on_k8s.client |
Examples |
| Retain a submitted run's identity | RunHandle, returned by SparkOnK8S.last_submitted_run |
Automation and recovery |
| Inspect startup or runtime problems | Diagnosis, returned by SparkOnK8S.diagnose_app() |
Diagnostics |
| Provision namespace access during setup | SparkOnK8SNamespaceSetup from spark_on_k8s.utils.setup_namespace |
Namespace setup |
A run handle identifies a submission; it does not certify data correctness. Observation timeouts and cancellation have different effects. Read the lifecycle rules in the linked guides before automating retries or cleanup.
Job definitions and integrations¶
| Task | Entry point or result | Usage guide |
|---|---|---|
| Load a job document and select a profile | JobSpec from spark_on_k8s.jobs |
Job files |
| Inspect an offline submission plan | ApplicationPlan, returned by planning methods |
Planning a job |
| Run a Spark task through Airflow | SparkOnK8SOperator from spark_on_k8s.airflow.operators |
Airflow lifecycle and permissions |
| Connect to an existing Spark Connect endpoint | ConnectProfile from spark_on_k8s.connect |
Spark Connect |
| Create a private Spark Connect server | ConnectServerConfig and ConnectServerHandle from spark_on_k8s.connect_server |
Server lifecycle |
SQL submission uses the client's submit_sql() method or the
Airflow operator's sql / sql_file options. See SQL jobs
for the required extra and execution behavior.
Implementation reference¶
The Implementation details navigation retains module pages for contributors
and existing links, including private helpers and optional integration internals.
It also includes functions defined in package __init__.py files, such as the
SQL preparation helpers and
API client holders. Empty package markers are omitted.
A generated signature is not a support promise for every importable object. The support and compatibility policy defines which documented contracts receive migration guidance. For CLI commands, HTTP routes, configuration and deployment behavior, use the corresponding user guide rather than importing their implementation functions.