Remote job submission¶

Remote submission lets an automation runner send an approved SparkJob to the API without loading a local kubeconfig. It is disabled by default. Enabling it also requires bearer authentication for the existing API, dashboard, static assets, proxies and WebSockets. Only the GET/HEAD liveness (/health) and readiness (/ready) probes remain public.

This is an operator API, not a multi-tenant authorization system. Its bearer token grants access to the existing read and management routes as well as admitted submissions. The submission policy restricts new jobs; it does not restrict an authenticated operator's existing list, logs, kill or delete permissions. Use a separate deployment and Kubernetes service account for each trust boundary.

Configure the server¶

Install the api extra and configure the server's Kubernetes credentials as usual. Requests cannot supply a kubeconfig, context, client configuration or cluster endpoint. Create a random token, store it in a restricted file or Kubernetes Secret, and mount a policy file. Do not commit either deployment credentials or tokens to source control.

export SPARK_ON_K8S_SUBMISSION_ENABLED=true
export SPARK_ON_K8S_SUBMISSION_TOKEN_FILE=/run/secrets/submission-token
export SPARK_ON_K8S_SUBMISSION_POLICY_FILE=/etc/spark-on-k8s/submission-policy.json
spark-on-k8s api start

The token must contain at least 32 printable ASCII characters without spaces; use a cryptographically random value such as one generated by secrets.token_urlsafe(32). During a rotation, the token file may contain two tokens on separate lines, and either is accepted. They are equivalent operator credentials, not separate identities. Missing or malformed configuration prevents startup.

Each API worker checks the token and policy files for changes at most once every SPARK_ON_K8S_SUBMISSION_RELOAD_INTERVAL seconds (default 10, at most 3600), when it handles a request. 0 reads the files only at startup. Changes are detected by file identity, size and modification time, which covers in-place edits, atomic renames and the symlink swap Kubernetes uses to update Secret and ConfigMap volumes. A change applies only when both files are valid and the policy's namespaces are within SPARK_ON_K8S_API_NAMESPACES; the worker logs a warning when it applies one. Otherwise it keeps its last valid token and policy and logs an error once for that change, without token values: a token you removed remains accepted until a valid change is loaded, so check the logs after each change. SPARK_ON_K8S_SUBMISSION_ENABLED, the file paths and the reload interval are read only at startup. With the Helm chart, set the remote values instead; see chart remote submission.

Terminate TLS at a trusted reverse proxy or gateway. Do not expose the backend's plain HTTP port outside that trusted boundary. For browser access, the gateway must authenticate the browser, protect write requests against CSRF, and inject Authorization: Bearer <server-token> on every HTTP and WebSocket request. Ordinary browser WebSockets cannot attach a custom authorization header themselves. The application does not accept query-string tokens or bearer cookies and does not place a token in the dashboard bootstrap. The gateway must remove client-supplied identity headers before injecting its own and keep the backend unreachable directly. With or without remote mode, the Spark UI and History Server proxies forward only content-negotiation, content-type, range and conditional-cache request headers. Bearer tokens, cookies, gateway identity headers and other credentials are not sent to application services. Log WebSockets use the server's Kubernetes client and do not forward browser authorization headers to pods. Standalone mode with remote submission disabled does not require bearer authentication. See browser and gateway security boundaries.

The Airflow task UI also makes server-to-server console requests. It does not have a bearer-token configuration option. When the Spark API uses remote mode, set SPARK_ON_K8S_AIRFLOW_SERVICE_URL to an administrator-controlled gateway that authenticates the Airflow service and injects the operator credential, or use a separate internal API deployment with remote mode disabled and access restricted to trusted services. Browser gateway injection alone does not authenticate the Airflow backend's requests. Never put the operator token in browser configuration, task arguments or the service URL.

Apply least-privilege Kubernetes RBAC, Pod Security admission, namespace ResourceQuotas and network policy independently of this feature. Approved code and images must be trusted: this API does not sandbox the application's code or prevent that code from using its service account. Set ingress request/concurrency limits for the capacity of the server and cluster.

Admission policy¶

This example uses an illustrative image digest; replace it with an image you own and have reviewed. Application code must already be present inside that image. Prefer immutable digests, because an exact mutable tag can point to different code later.

{
  "schema_version": 1,
  "namespaces": {"analytics": ["spark-worker"]},
  "applications": [
    {
      "image": "registry.example.com/data/report@sha256:1111111111111111111111111111111111111111111111111111111111111111",
      "app_path": "local:///opt/jobs/report.py",
      "class_name": null,
      "arguments": [[], ["2026-11-07"]]
    }
  ],
  "limits": {
    "driver_cpu": 2,
    "driver_memory_mb": 2048,
    "executor_cpu": 2,
    "executor_memory_mb": 2048,
    "executors": 4
  },
  "spark_conf": {"spark.sql.shuffle.partitions": [8, 16]}
}

Namespaces and service accounts are exact pairs. Each application entry binds an exact image, local application path, optional Java class and approved argument lists. Omitting arguments permits only an empty list. Arguments are not arbitrary free-form input: an approved wrapper could otherwise use them to execute unapproved code. Without the optional placement keys, an approved application may run with every approved namespace and service-account pair.

Memory ceilings include memory overhead. Driver and executor limits apply per process; the executor count also bounds dynamic allocation, which must declare an explicit maximum. The ordinary portable job defaults still apply: one CPU, 1024 MB plus 512 MB overhead per process, and two executors. A policy below those defaults requires explicit smaller job values.

Remote jobs may set spec.max_runtime (seconds), which makes Kubernetes stop the driver that long after its pod starts; it can only shorten a run. To bound runs whose remote client is lost or never stops them, an administrator can add the optional "max_runtime": 86400 to limits. Every remote job must then set spec.max_runtime no greater than that limit, or it is rejected; it is never filled in silently. Policies without this key keep their current behavior. See driver lifetime limits.

Remote requests reject pod templates, packages, secret imports, custom labels/annotations, node selectors, profiles, Kubernetes credentials and all other unsupported fields, except a policy's required labels. Only these Spark tuning keys can appear in the administrator's explicit value allowlist:

spark.sql.shuffle.partitions
spark.default.parallelism
spark.sql.adaptive.enabled
spark.sql.adaptive.coalescePartitions.enabled
spark.sql.autoBroadcastJoinThreshold
spark.sql.broadcastTimeout
spark.sql.session.timeZone

Values must match the policy, including their JSON types. Kubernetes settings, classpaths, plugins, environment variables, remote files, resource overrides and arbitrary Spark keys cannot be enabled through that allowlist. The API validates the generated pod before writing resources, and uses only the server-configured client for both planning and submission.

Restrict where an application runs¶

The optional application keys namespaces and service_accounts and the optional top-level labels extend schema_version: 1. Policies without them are admitted exactly as before. An API version that does not support these keys refuses to start with a policy that uses them, instead of ignoring the restriction.

{
  "schema_version": 1,
  "namespaces": {
    "analytics": ["spark-worker"],
    "finance": ["spark-worker", "spark-finance"]
  },
  "applications": [
    {
      "image": "registry.example.com/data/report@sha256:1111111111111111111111111111111111111111111111111111111111111111",
      "app_path": "local:///opt/jobs/report.py",
      "namespaces": ["analytics"]
    },
    {
      "image": "registry.example.com/finance/ledger@sha256:2222222222222222222222222222222222222222222222222222222222222222",
      "app_path": "local:///opt/jobs/ledger.py",
      "namespaces": ["finance"],
      "service_accounts": ["spark-finance"]
    }
  ],
  "limits": {
    "driver_cpu": 2,
    "driver_memory_mb": 2048,
    "executor_cpu": 2,
    "executor_memory_mb": 2048,
    "executors": 4
  },
  "labels": {
    "required": {
      "cost-center": {"values": ["cc-1001", "cc-1002"]},
      "example.com/team": {"pattern": "[a-z][a-z0-9-]{1,30}"}
    },
    "fixed": {"submitted-via": "remote-api"}
  }
}

A job's namespace and service account must still be an approved pair. When the matched application entry lists namespaces or service_accounts, the job must also use one of the listed values, so the report above can run only as analytics/spark-worker and the ledger only as finance/spark-finance. Several entries may approve the same code with different placements; a job is admitted when any matching entry allows its pair. Otherwise it is denied with policy_denied and the message The application is not approved for this namespace and service-account pair. Every listed namespace must be a policy namespace, and every listed service account must be approved for one of the entry's namespaces, or the policy is rejected.

Placement restricts which approved code can run with which Kubernetes identity. It does not identify the caller: every bearer can still submit every approved application to its approved placements.

Require cost-attribution labels¶

labels.required names the driver labels every remote job must set in spec.driver_labels:

spec:
  driver_labels:
    cost-center: cc-1001
    example.com/team: data-eng

A rule lists exact values, or a pattern (a Python regular expression that must match the whole value), or is empty ({}) to accept any value. Values must be nonempty Kubernetes label values. Jobs can set only the required labels: other driver labels, executor labels and annotations remain rejected, and a policy without required rejects driver_labels as before. The server adds the labels.fixed labels to every remote job; jobs cannot set them. Admitted and fixed labels are applied to the driver pod and, through spark.kubernetes.executor.label.*, to its executors, so cost tools can attribute both.

The policy cannot require or fix reserved labels: unprefixed keys beginning with spark- (such as spark-role and spark-app-id), and keys under the spark-on-k8s.io/ prefix, used by the Spark Connect and Airflow integrations, or under kubernetes.io/ and k8s.io/ and their subdomains. A remote job can therefore never set or replace the labels that identify and own its driver. Labels support cost attribution; they are not an authorization or tenant boundary.

Submit from the CLI¶

The remote CLI needs only the core package and a YAML or JSON job file. It does not require the api extra, Airflow, or local Kubernetes credentials. The job's namespace, service account, image, path, class, arguments and resources must pass the server policy.

export SPARK_ON_K8S_API_URL=https://spark.example.com/team
export SPARK_ON_K8S_API_TOKEN_FILE=/run/secrets/spark-api-token
spark-on-k8s remote plan report.yaml
spark-on-k8s remote submit report.yaml --timeout 30

--api-url and --token-file override those variables. SPARK_ON_K8S_API_TOKEN is an alternative to a token file; prefer the file in shared environments. Tokens are never accepted as a command line argument. HTTPS certificate verification is mandatory. Plain HTTP is accepted only for loopback development. Redirects are refused so credentials cannot be forwarded to another location, and environment HTTP proxies are not used by this client.

Output defaults to one versioned JSON object on stdout. --output text prints a redacted plan or the submitted driver pod name. A successful submission returns HTTP 201 and this shape:

{
  "schema_version": 1,
  "operation": "submit",
  "run": {
    "namespace": "analytics",
    "app_id": "report-0123456789abcdef0123456789abcdef",
    "driver_pod": "report-0123456789abcdef0123456789abcdef-driver",
    "driver_uid": "5bdceeb0-7fa8-4f72-823b-80f26e6195c4"
  },
  "status": null,
  "error": null
}

The readable name prefix is shortened when necessary to leave room for the random run identity and Kubernetes driver suffix. Each request creates a distinct run; a plan does not reserve its preview identity. Submission returns after resource creation and does not wait for Spark to finish.

The CLI uses the automation exit codes: 0 for an accepted operation, 1 for a remote/API/transport failure, 2 for invalid local input, 4 for a missing/disabled remote route, 5 for a request timeout and 130 for interruption. Errors have schema_version, operation and error: {code, message}; command-line parsing errors still use Click's normal stderr output.

HTTP contract and retry safety¶

Send the SparkJob document directly as JSON to POST /jobs/v1/plan or POST /jobs/v1/submit, with the bearer header. Both use the same admission rules. Plans return HTTP 200 with {schema_version: 1, operation: "plan", plan: {...}, error: null} and redact application arguments. Requests are limited to 1 MiB, and receiving the body has a 15-second deadline.

The remote CLI bounds responses to 4 MiB and applies a 30-second HTTP deadline by default, configurable up to 300 seconds. Stalled response headers or bodies are interrupted. Hostname resolution also depends on the operating system's resolver timeout. This timeout does not cancel a server operation or stop a Spark job.

Requests are never retried automatically. A timeout, disconnect or Kubernetes failure can occur after some or all resources were created. Inspect existing runs before retrying to avoid duplicates; partial resources may require operator cleanup. The CLI also reports invalid_response with this warning when a submission receives a success or 5xx status whose body is not a valid result: the server or a gateway may have accepted the job. Idempotency-Key is explicitly rejected until a durable, shared idempotency implementation is available. In-memory deduplication would not be reliable across API workers or replicas.

Authentication failures return 401, admission denials 403, disabled routes 404, unsupported idempotency or invalid JSON 400, a client that disconnects before its body arrives 400 client_disconnected (recorded only: the client is gone), bodies not received within the deadline 408, oversized bodies 413, unsupported content types 415, and invalid job schemas 422. Kubernetes conflicts return 409; other Kubernetes failures return 502, and unexpected server failures 500. Errors contain a stable code and a human-readable message, never the Kubernetes response body. Every response carries an X-Request-ID header. The API's audit events record each plan and submission with that ID, its outcome and, once admitted, the generated run identity.

Admission limits apply to each submitted job. They do not reserve shared capacity or account for concurrent accepted jobs. Enforce aggregate resources with Kubernetes ResourceQuota or your external admission system. A policy rejection, a quota rejection and a scheduler wait are different outcomes.

Namespace allowlists constrain the shared operator deployment. Every bearer holder retains management access within that deployment's exposed surface; the allowlist does not establish individual user or tenant identity. Use separate trust boundaries where independent authorization is required.

Rotate the token¶

A rotation keeps a short overlap in which the current and the new token are both accepted:

Add the new token as a second line of the server's token file.
When every API pod has the updated file, switch every client to the new token: CLI token files, automation secrets and any gateway that injects the bearer header.
Replace the server's token file with the new token alone. Once that update reaches every pod, the old token receives 401.

With the Helm chart's remote values, the chart mounts the Secret's token and policy keys as a directory under remote.mountPath (/etc/spark-on-k8s/remote by default), so Secret updates reach running pods. If you configure remote mode through the generic env, volumes and volumeMounts values instead, also mount the Secret as a directory. Do not mount these files with subPath: such mounts never receive Secret updates, so every step would need a pod restart. Rotate with files in a private directory, so tokens never appear in command lines. Recreating the Secret replaces both keys, so copy the deployed policy first:

umask 077
namespace=spark-system
secret=spark-api-remote
kubectl get secret "$secret" --namespace "$namespace" --output jsonpath='{.data.token}' \
  | base64 --decode > current-token
kubectl get secret "$secret" --namespace "$namespace" --output jsonpath='{.data.policy\.json}' \
  | base64 --decode > policy.json
# No trailing newline: the chart's ServiceMonitor sends the token file byte for byte.
python3 -c 'import secrets; print(secrets.token_urlsafe(32), end="")' > next-token

# Step 1: accept both tokens. Blank lines are ignored; the newline must separate the tokens.
{ cat current-token; echo; cat next-token; } > overlap-token
kubectl create secret generic "$secret" --namespace "$namespace" \
  --from-file=token=overlap-token --from-file=policy.json=policy.json \
  --dry-run=client --output yaml | kubectl apply --filename -

The kubelet projects Secret updates into running pods after its sync period, typically within a minute or two. Before step 2, confirm that each API pod sees the new file by comparing a checksum rather than printing tokens; for example, run kubectl exec --namespace "$namespace" <api-pod> -- sha256sum /etc/spark-on-k8s/remote/token for every pod and compare it with the checksum of overlap-token. A worker checks the files at the first request it handles once SPARK_ON_K8S_SUBMISSION_RELOAD_INTERVAL seconds have passed since its previous check, and logs the number of accepted tokens when it applies the change. An error in the log means that worker still uses the previous token and policy.

# Step 3, after every client uses the new token: accept only the new token.
kubectl create secret generic "$secret" --namespace "$namespace" \
  --from-file=token=next-token --from-file=policy.json=policy.json \
  --dry-run=client --output yaml | kubectl apply --filename -

With the chart's ServiceMonitor, metric scrapes fail from step 1 until step 3 reaches the pods, because Prometheus cannot send a two-line token. Store the new token in your secret manager and delete the local files afterwards. Policy changes reach the API the same way. To revoke a compromised token without waiting for the kubelet, update the Secret and restart the API pods; restarted workers read the current files at startup and refuse to start if they are invalid.