Operational boundaries¶

Know which identity is acting¶

The local client uses its selected kubeconfig/context; the API submits with its own Kubernetes identity. Driver and executor service accounts govern workload access to Kubernetes and external storage. An Airflow connection selects the submitter identity and is separate from the task UI's service credentials. A gateway authenticates callers; it does not make a shared API service account act as each caller. Use separate API deployments and namespaces for separate trust boundaries. Do not treat namespace filtering as tenant authorization.

The convenience namespace setup helper currently creates a cluster-scoped binding to Kubernetes' edit role. Administrators should provision a namespace-scoped RoleBinding for production deployments instead. Neither the chart nor this library is a cloud IAM administrator. Prefer workload identity and external Secret references to literal credentials in job files.

Diagnose evidence before changing resources¶

Record the cluster context, namespace, application ID and driver UID with your incident. Distinguish application fetch time, diagnostic collection time and Kubernetes event time. Refresh before acting on an old view. Retained history can say Running after an application disappears; it is not current health. Incomplete reads, missing events and empty logs do not prove health or justify resubmission. Compare current and previous container termination reasons, scheduler events and driver/executor logs before increasing resources.

Use namespace-scoped commands, substituting only identifiers from the receipt:

kubectl --context CONTEXT -n NAMESPACE get pod DRIVER -o wide
kubectl --context CONTEXT -n NAMESPACE describe pod DRIVER
kubectl --context CONTEXT -n NAMESPACE logs DRIVER -c driver
kubectl --context CONTEXT -n NAMESPACE get events --field-selector involvedObject.uid=DRIVER_UID

Do not replay data writes, delete shared PVCs or repair streaming checkpoints based only on a pod failure. Keep images, job definitions and external data recovery procedures independently versioned.

Bound and clean up abandoned runs¶

A killed client, CI runner or worker cannot stop what it submitted. For runs with a known upper duration, set max_runtime so Kubernetes stops the driver itself. Finished drivers keep their logs, Services, ConfigMaps and Secrets until deleted. Review spark-on-k8s apps prune --namespace NAMESPACE --older-than SECONDS before adding --delete: deleting a driver also removes its logs and owned objects, so collect the evidence above first. Active drivers are only selected with --include-active.

Back up optional history consistently¶

The SQLite history store is an operational index, not a job queue or durable source of submission truth. Restrict database and backup access because labels and operational metadata may be sensitive. A live database may use WAL files; copying only the .sqlite3 file while it is being written is not a safe backup. Use SQLite's online backup API (or stop all writers before a filesystem copy):

import sqlite3
from pathlib import Path

source = Path("/var/lib/spark-on-k8s/runs.sqlite3").resolve()
backup = Path("/secure-backups/runs.sqlite3").resolve()
with sqlite3.connect(source.as_uri() + "?mode=ro", uri=True) as original:
    with sqlite3.connect(backup) as target:
        original.backup(target)
        assert target.execute("PRAGMA integrity_check").fetchone() == ("ok",)

Choose a new protected destination and test restoration with the exact package version in a disposable deployment. Confirm records and retention settings, then verify live Kubernetes state separately. There is no automatic migration, restore endpoint or claim of multi-writer/high-availability database support.

Qualify an additional integration¶

Cloud identity, table formats, accelerators and new orchestrators need a named owner and a reproducible acceptance test before joining the supported matrix. Pin image and connector versions. Prove work runs in executors, verify actual output and a controlled failure, and exercise interruption/restart semantics. For storage, check namespace, mount ownership and persistence across driver replacement. For streaming/lakehouse jobs, distinguish pod recovery from checkpoint correctness, table commits, ordering and replay guarantees.

Prefer a small recipe using the existing SDK and receipts over another control plane; see the ecosystem recipes. The Spark Operator assessor reports unsupported source fields; it does not silently translate unknown Spark Operator semantics. OpenLineage reports and Airflow run correlation are optional evidence, not proof that every data write committed. Deployment examples do not establish arbitrary multi-cloud or multi-tenant isolation guarantees.

Audit events and metrics¶

The API writes one structured audit event for each remote plan or submit, kill and delete request, and for each request refused by bearer authentication or the namespace scope. An event identifies a request: its route, target, outcome and the address it came from. It does not identify a person. The API has no user identity, and the shared operator bearer is the same for every caller. Run history is a polled metadata index and is not an audit trail either.

Every HTTP response carries an X-Request-ID header with a new random ID generated by the server. The API ignores an X-Request-ID sent by the client. To correlate with gateway logs, have the gateway log the upstream response header.

Configure the audit log¶

Variable	Default	Meaning
`SPARK_ON_K8S_AUDIT_LOG_ENABLED`	`true`	Write audit events to the `spark_on_k8s.audit` logger; `true` or `false`.
`SPARK_ON_K8S_AUDIT_IDENTITY_HEADER`	empty	A request header that a trusted gateway sets, such as `X-Auth-Request-User`, recorded as `unverified_identity`.

The log is enabled by default, because an event has to be recorded before you need it. Events are written only for the audited operations and for refusals, so each refusal adds one line next to the access-log line the server already writes for it. Invalid values prevent startup. Credential headers such as Authorization and Cookie cannot be used as the identity header.

spark-on-k8s api start writes each event to stdout as one JSON object per line. Each worker process writes its own events. Unlike the server's other logs, these lines do not depend on --log-level. If the deployment already configures logging handlers for spark_on_k8s.audit or its parent loggers, for example when another server embeds the application, the API sends the events to those handlers instead.

Event schema¶

Events have audit_version: 1. A compatible version can add fields. A change to or removal of an existing field increments the version.

Field	Value
`time`	UTC time, to the millisecond, when the request finished.
`event`	`job.plan`, `job.submit`, `app.kill`, `app.delete`, or `access` for a refused request to any other route.
`outcome`	`succeeded`, `denied`, `rejected`, `failed` or `unknown`; see below.
`code`	`ok`, or the `error.code` from the response's error body.
`status`	The HTTP status. A refused WebSocket handshake is recorded as `403`.
`request_id`	The value of the `X-Request-ID` response header.
`method`	The HTTP method, or `WEBSOCKET`.
`route`	The route template, such as `/app/{namespace}/{app_id}`, never the raw path or query string. Authentication happens before routing, so a request it refuses has a `null` route unless it targets an audited operation.
`client`	The client address that the ASGI server reports; see the limits below.
`unverified_identity`	The configured gateway header, if the request has exactly one; otherwise `null`.
`namespace`, `app_id`, `driver_pod`, `driver_uid`	The target, when known; otherwise `null`.

Outcomes describe what the API did, not what the Spark application did:

succeeded: the API completed the request. A submission created the driver resources and a kill sent the termination signal. Neither means the Spark work succeeded.
denied: HTTP 401 or 403. The request was refused by authentication, the namespace scope or the submission policy, or Kubernetes RBAC refused the API's service account (code forbidden).
rejected: another 4xx response, such as invalid_job, application_not_found, application_state, invalid_request for an identifier Kubernetes refused, or client_disconnected for an upload the client abandoned. The API changed nothing.
failed: a server error during a plan, which creates nothing.
unknown: a server or Kubernetes failure during a submit, kill or delete. The resources may have been created, signalled or deleted anyway. An uncertain result does not mean nothing happened: inspect the recorded run before retrying.

After a submission is admitted, its event records the generated namespace, app_id and driver_pod, even if Kubernetes fails afterwards. It records driver_uid once the driver pod exists. For denied or invalid documents, these fields are null. A plan records only its namespace, because the identity in a plan is illustrative and no run exists. Kill and delete events record the target from the request path. access events record the refused namespace and any application identifiers in the path.

{"audit_version":1,"time":"2026-11-09T10:15:02.418Z","event":"job.submit","outcome":"succeeded","code":"ok","status":201,"request_id":"6f1c2a9e0b5d4e7f8a3b2c1d0e9f8a7b","method":"POST","route":"/jobs/v1/submit","client":"10.42.0.17","unverified_identity":"alice@example.com","namespace":"analytics","app_id":"report-0123456789abcdef0123456789abcdef","driver_pod":"report-0123456789abcdef0123456789abcdef-driver","driver_uid":"5bdceeb0-7fa8-4f72-823b-80f26e6195c4"}
{"audit_version":1,"time":"2026-11-09T10:16:40.007Z","event":"app.kill","outcome":"unknown","code":"internal_error","status":500,"request_id":"0d8e4f1a2b3c4d5e6f708192a3b4c5d6","method":"PATCH","route":"/app/{namespace}/{app_id}","client":"10.42.0.17","unverified_identity":null,"namespace":"analytics","app_id":"report-0123456789abcdef0123456789abcdef","driver_pod":null,"driver_uid":null}
{"audit_version":1,"time":"2026-11-09T10:17:11.250Z","event":"access","outcome":"denied","code":"namespace_forbidden","status":403,"request_id":"9a8b7c6d5e4f40312a1b0c9d8e7f6a5b","method":"GET","route":"/webserver/ui/{path:path}","client":"10.42.0.17","unverified_identity":null,"namespace":"kube-system","app_id":null,"driver_pod":null,"driver_uid":null}
{"audit_version":1,"time":"2026-11-09T10:18:30.912Z","event":"access","outcome":"denied","code":"unauthorized","status":401,"request_id":"47c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5","method":"GET","route":null,"client":"10.42.0.9","unverified_identity":null,"namespace":null,"app_id":null,"driver_pod":null,"driver_uid":null}

Events never include these values: authorization or cookie headers, bearer tokens, query strings, request or response bodies, job documents (including arguments, images and Spark configuration), Kubernetes response bodies, or exception text. Names in the target fields come from the workload or the request and can themselves be sensitive. Restrict access to these logs as you would for other API logs.

Limits of the audit log¶

Logs are the only transport. Collecting the logs, sending them off the node, retaining them, controlling access and protecting their integrity are the job of your log pipeline. Events are written as each request finishes. They can be lost if the process stops or the node fails before collection. The log is not tamper-evident and is not a durable audit store. A failure to write the log does not fail the request.
The client address is the network peer. client is the address the ASGI server reports. Behind a gateway or ingress, this is usually the gateway's address. The server reports another address only when its trusted-proxy setting accepts X-Forwarded-For from that peer. Uvicorn, which spark-on-k8s api start runs, accepts these headers only from the addresses in FORWARDED_ALLOW_IPS (default 127.0.0.1). Keep that list limited to your gateways.
The gateway identity is not verified. The API records the configured header but does not authenticate it. The value is meaningful only if the gateway authenticates users, removes any copy of the header sent by the client, and the backend cannot be reached except through the gateway. If the header appears more than once, the API records null. Every bearer holder has the same permissions: the recorded identity helps attribute a request, but does not authorize it.
Kubernetes sees the API. Kubernetes audit logs record the API's service account as the actor. To find the matching API event, compare the time, namespace and application ID.
WebSocket handshakes and responses that the ASGI server writes itself, such as its concurrency-limit 503, do not carry X-Request-ID.

Operation metrics¶

When metrics are enabled (SPARK_ON_K8S_METRICS_ENABLED=true), /metrics also serves these counters:

spark_on_k8s_api_audit_events_total{event,outcome,code}: increases by one for each audit event, including when the audit log is disabled.
spark_on_k8s_api_kubernetes_errors_total{status_class}: counts Kubernetes API error responses (4xx, 5xx, or other for a response without a usable status) received while serving kill, delete, submit, console, log-stream and application list requests. Readiness probes are not counted. The run-history collector reports its own spark_on_k8s_collection_failures_total. Connection failures and timeouts that return no Kubernetes response are not counted here. They appear as failed or unknown outcomes.

Label values come only from the fixed event and outcome names and the API's own error codes. Labels never contain a namespace, application ID, route or client. A namespace label would have no bound with the * scope, and refused namespaces are chosen by the caller, so any value in a request could create a new series. Use the audit log for per-namespace detail. A series appears after its first event.

The counters are kept in the memory of each worker process and reset when that worker restarts. The API image runs four workers by default (SPARK_ON_K8S_API_WORKERS). Each scrape reaches one of them, so successive samples can come from different workers, and rate() can mistake the differences for counter resets. If you alert on these counters, run one worker per pod (SPARK_ON_K8S_API_WORKERS=1). Alternatively, count the audit lines in your log pipeline, which receives the events of every worker. For example, with one worker:

sum by (code) (increase(spark_on_k8s_api_audit_events_total{outcome="denied"}[10m]))
increase(spark_on_k8s_api_audit_events_total{event="job.submit",outcome="unknown"}[15m]) > 0
increase(spark_on_k8s_api_kubernetes_errors_total{status_class="5xx"}[5m]) > 0

Authorization review matrix¶

Surface	Authenticator / authorizer	Effective cluster identity	Actor attribution
Local SDK/CLI	Kubernetes credentials and RBAC	Selected local or in-cluster identity	Kubernetes audit configuration
Standalone internal API	External network/gateway controls	API service account	No built-in end-user principal
Remote-enabled API and metrics	Shared operator bearer and admission policy	API service account	Shared operator, not a tenant identity
Airflow task tab	Airflow session, task permissions, ownership checks	Configured Spark API service identity	Airflow task binding; separate API has its own controls
Airflow DAG execution	Airflow Kubernetes connection and cluster RBAC	Worker/triggerer connection identity	Task execution and Kubernetes audit configuration

Disabling Airflow tab writes only disables browser mutations; it does not prevent DAG tasks from submitting workloads. Gateway TLS alone does not establish all these boundaries. See browser and gateway checks for prefixes, static assets, HTTP, WebSockets and direct-backend isolation.

Recovery restores recorded observations through the backup point, then collection resumes from currently visible pods. It cannot reconstruct deleted pods or intermediate statuses during an outage. Readiness indicates current service health, not historical completeness. Diagnostics classify infrastructure evidence, not workload efficiency; neither a successful pod nor an empty finding list means a Spark job is tuned.