Skip to content

Release notes

Published GitHub release notes and the next release in preparation live on this permanent page, independently of the documentation version selected elsewhere on this site. They cover the spark-on-k8s Python package and its API image; the Helm chart and the experimental dbt adapter have their own version numbers, which a release mentions when it ships them.

Releases marked In preparation are drafts. Their installation commands and release links become available when the release is published.

Spark On K8S 0.13.0

Published 2026-11-11 · GitHub release

Spark On K8S 0.13.0 is the largest release so far. It supports Airflow 3 next to Airflow 2.11 and Spark 4 next to Spark 3.5, and replaces the web UI with a shared React console that also runs inside the Airflow 3 task view. It adds portable job files, standalone SQL jobs, Spark Connect profiles and private Connect servers, diagnostics, run history, authenticated remote submission, API audit events and an optional limit on how long a driver may run. It also fixes many submission, CLI, API and Airflow lifecycle bugs. Python 3.11 or newer is now required.

python -m pip install 'spark-on-k8s==0.13.0'

Read the 0.13.0 documentation and its installation guide before upgrading from 0.12.1. The API image is published as ghcr.io/hussein-awala/spark-on-k8s:0.13.0 and husseinawala/spark-on-k8s:0.13.0.

Significant changes and upgrade notes

  • Python 3.11 or newer is required; Python 3.9 and 3.10 support is removed. The core client and API are tested on Python 3.11–3.15. The airflow, airflow-ui and connect extras require Python below 3.15. #180, #181, #182
  • Airflow 3 is supported, and the minimum Airflow version is now 2.11.2. The airflow extra requires apache-airflow-providers-cncf-kubernetes 10.25 or newer, and the core package requires the kubernetes client 36.x to match it. Use Airflow 3 for new deployments; the native task tab needs Airflow 3.4+ and the airflow-ui extra. Both Airflow extras install under Airflow 3.4.0's constraints files; Airflow 2.11.2's constraints pin older kubernetes, kubernetes-asyncio, provider and sqlparse versions, so override those pins there. #179, #180, #189, #269
  • Failures are now reported as failures. CLI commands that wait for an application exit non-zero when the Spark workload fails or the wait times out, and offer a versioned JSON result for automation. Spark SQL tasks fail on the first failed statement instead of logging the error and succeeding. Review scripts that relied on a zero exit code. #162, #190, #194
  • Deferrable Airflow tasks enforce startup_timeout. A driver that stays Pending past the deadline is deleted and the task fails instead of waiting indefinitely. Recovered tasks now reattach to a Pending driver instead of submitting a duplicate. #168, #172
  • Configuration values are parsed as documented. SPARK_ON_K8S_IN_CLUSTER=false no longer selects in-cluster authentication, and only true (any case) selects it, so 1, yes and on now mean kubeconfig authentication; SPARK_ON_K8S_APP_WAITER=log now streams logs, and configured application arguments and dictionary defaults are no longer dropped by the CLI. Check environments that set these variables. A value that cannot be parsed raises ConfigurationError naming the variable, and the CLI prints one error line instead of a traceback, for every command. #159, #160, #166, #174, #280
  • Executor volumes now use Spark's option keys. hostPath, NFS and PVC executor volumes are mounted, and emptyDir settings are applied. Previously they were emitted with the wrong keys. #155
  • The web UI is a new React console. The old server-rendered pages are replaced. Static assets are packaged in the wheel and no CDN is needed. Deployments mounted under a URL prefix keep working through the Spark UI and History Server proxies. #188, #202, #206
  • Long application IDs get hashed driver names, and unusable IDs fail early. Driver Pods for application IDs longer than 56 characters are named with a hash of the ID instead of a truncated prefix, so reruns no longer collide. Generated ConfigMap names follow the same rule. IDs that cannot name the driver Service, such as suffixes with dots or uppercase letters, are rejected before any resource is created. #221, #228
  • The API reports target-state errors with their status. Stopping or deleting an application now returns 403, 404 or 409 instead of 500, with an error code. An unreachable Spark UI or History Server returns 502 or 504. #220
  • API errors share one JSON envelope. Every error response, including unknown routes, invalid parameters and unexpected failures, is {"schema_version": 1, "error": {"code", "message"}}, the shape /jobs/v1 already used. Clients that read detail must read error.code or error.message; unexpected failures return internal_error with a logged reference instead of plain text. job validate --output json adds operation and a top-level error, and reports each failing profile as {code, message}. #234
  • The console's Kill and Delete button settings ignore letter case, like other true/false settings. True now shows the buttons; previously only lowercase true did. These settings do not restrict the API routes. #226
  • The API serves only its own namespace by default. SPARK_ON_K8S_API_NAMESPACES selects the namespaces the API, its Spark UI proxy, driver log streams and console serve: empty for the current namespace, * for every namespace, or a comma-separated list. Other namespaces return 403 namespace_forbidden; previously the Spark UI proxy could reach any namespace's Spark UI. A standalone spark-on-k8s api start now serves the kubeconfig context's namespace: set SPARK_ON_K8S_API_NAMESPACES to keep reaching others. The API refuses to start when its default namespace, run-history namespaces or submission-policy namespaces fall outside the scope, so check SPARK_ON_K8S_API_DEFAULT_NAMESPACE before upgrading. The console offers a namespace picker for a listed scope. #232
  • The API image runs as a non-root user. The server runs as UID and GID 10001 instead of root, and chart 0.4.0 sets a matching security context that meets Pod Security restricted. Mounted files the API reads or writes must be accessible to that user or group. A kubeconfig mounted under /root is no longer readable: mount it at /home/spark-on-k8s/.kube/config or set SPARK_ON_K8S_CONFIG_FILE. See the chart guidance for ports, root-written volumes and opting out. #237
  • The API image runs Python 3.14 instead of 3.11. Its python:3.14-slim base clears the image-scan alerts that have no Python 3.11 fix, and the build installs pending Debian security updates, such as the patched zlib1g. Packages added in images built FROM the API image must support Python 3.14. #285
  • The API image no longer contains pip, setuptools or wheel. The build removes these packaging tools after installing, because the API does not use them and they carried most of the image's fixable advisories. Images built FROM the API image must run python -m ensurepip before pip install, or copy packages from a build stage; see the chart guidance. #261
  • The API writes audit events to its log by default. Remote plans and submissions, stop and delete requests, and requests refused by the token or the namespace scope each add one JSON line on stdout (spark_on_k8s.audit logger), and every HTTP response carries a server-generated X-Request-ID. The events identify requests, not people, and are not a durable audit store. Set SPARK_ON_K8S_AUDIT_LOG_ENABLED=false to turn them off. spark-on-k8s api start also sends the package's own log messages through the server's log handler at --log-level. #246, #253, #263
  • Browser log streams and the Spark UI proxy are checked. Driver-log WebSockets accept only a browser origin that matches the address the API sees; behind a TLS-terminating gateway, set SPARK_ON_K8S_API_WEBSOCKET_ALLOWED_ORIGINS to the public origin. The Spark UI and History Server proxies forward only content-negotiation, cache and range headers, so Authorization, cookies and identity headers no longer reach Spark UIs. #196, #200
  • Waits end on Kubernetes API errors. While the SDK, CLI or an Airflow task waits, an error other than 404 now ends the wait: 401 and 403 at once, throttling (429), server errors and lost connections after five consecutive failed reads. 0.12.1 ignored them once it had read a status. For deferred Airflow tasks, delete and keep_if_failed then delete the application; see the Airflow upgrade checklist. #164, #270
  • A driver that disappears during a wait is an unknown outcome, not a finished one. wait_for_app raises SparkAppDeletedError instead of returning, so submit_app(app_waiter="wait") raises too. The CLI reports not_found (exit code 4, status: null) as before, and an Airflow task fails without log replay or cleanup. #280
  • submit_app logs instead of printing. Without should_print=True, driver logs and status lines go to Python logging at INFO instead of stdout. Configure logging, or pass should_print=True, where a script read them from stdout. #149
  • Failed Airflow tasks log a postmortem by default. A failed, startup-timed-out or killed task logs a bounded diagnosis and the driver's last 100 log lines (secret-like values redacted) before cleanup, and pushes the XCom spark_failure_diagnosis. Set failure_log_tail_lines=0 where driver logs must not reach Airflow logs, or failure_diagnostics=False. #252, #274
  • Smaller behavior changes. The log viewer and its WebSocket start with the last 1,000 lines (at most 10,000) instead of the whole log. submit_app(secret_values={}) no longer falls back to SPARK_ON_K8S_SECRET_ENV_VAR. driver_labels can no longer override spark-app-name, spark-app-id or spark-role, and a user spark-ui-proxy label is dropped. Generated application ID suffixes use UTC instead of local time. #188, #191, #192, #278
  • Use Helm chart 0.4.0 with this API image. The chart's readiness probe uses /ready, which API 0.12.1 does not provide. Chart 0.4.0 also changes RBAC: names change, namespaces defaults to the release namespace, and a cluster-wide install binds a chart-owned ClusterRole instead of the built-in view or edit. Some 0.3.0 values no longer render: namespaces: "*" with readOnly: false needs rbac.allowClusterWideWrite: true, sparkHistory.enabled needs sparkHistory.image.tag or a digest, and image.tag must be a string (quote "0.12"). Read the migration guidance before upgrading from chart 0.3.0. The chart's Roles grant only the verbs the API calls: get and list on pods, get on pod logs and list on events, plus create and delete on pods, pod exec and Service create when readOnly is false; grant more through external RBAC if other tools share the account. #143, #157, #165, #232, #249, #282

Features

  • Run Spark 4 applications. Spark 3.5.9, 4.0.5 and 4.3.0 images are tested on real Kubernetes clusters in CI, and example images and Scala dependencies can be configured across the Spark 4 boundary. #193
  • Use the new web console to search, filter and page through applications, switch namespaces, inspect drivers, and stop or delete them with confirmation. The log viewer adds search, copy and download, tail selection, reconnection and paused scrolling. #188, #204, #205, #208
  • Open the same console from a native Airflow 3 task tab. Requests are checked against Airflow permissions, task identity, namespace and application ownership. The tab also works for operator subclasses and mapped tasks. #189, #203, #207, #250
  • Airflow retries reattach to the running driver of an earlier attempt, found by a task-key label on every driver, instead of submitting another application, including for dynamically mapped tasks. reattach_succeeded=True also adopts a succeeded driver as the attempt's result. #245
  • Describe jobs in versioned, portable job files with profiles. Use spark-on-k8s job validate, job plan and job submit, the Python SDK or either Airflow version. Offline plans never contact Kubernetes and redact secrets. #192
  • Reuse driver pod templates for sidecars, security settings, volumes and scheduling, and executor pod templates from inline YAML or a file. Set driver environment variables with spark.kubernetes.driverEnv.* properties. #175, #191, #216
  • Run SQL files or inline statements without Airflow through SparkOnK8S.submit_sql and spark-on-k8s sql submit. Statements run in order, stop on failure, and keep semicolons inside strings and comments. #173, #194
  • Track runs with an immutable RunHandle, durable submission receipts, and UID-guarded app cancel. Existing submit_app callers still receive the same value, the driver Pod name. #190, #200
  • Connect to existing Spark Connect servers through named profiles with spark-on-k8s connect check, or create a private server with connect server create|status|wait|delete|list|plan. connect check and connect server wait warn when the client and server Spark versions differ. Private servers accept image pull Secrets, Secret-backed environment variables, node selectors and tolerations. Tested Jupyter notebooks show the full workflow, and a guide builds an ML image. #195, #209, #210, #242, #258
  • Diagnose Pending or failed applications with spark-on-k8s app diagnose, the console or the Airflow task tab, and check cluster prerequisites with spark-on-k8s doctor. Findings cover image pulls, scheduling, permissions, OOM kills, crash loops and unavailable volumes. #198
  • Keep an optional, metadata-only record of runs after their driver pods disappear, with readiness and metrics endpoints. One stale namespace degrades readiness instead of failing it (SPARK_ON_K8S_RUN_HISTORY_READINESS=strict restores the all-namespaces rule), and the console warns about stale records. #199, #246
  • Let remote users plan and submit administrator-approved jobs through the API with spark-on-k8s remote plan|submit, gated by an operator token and an explicit allowlist policy. A policy can limit each application to namespaces and service accounts, require or add labels, and require a runtime limit. Token and policy files are reloaded without a restart, and two tokens are accepted during a rotation. #196, #244, #255
  • Start from tested ETL and Iceberg starter projects. #197
  • Limit how long a driver may run with max_runtime: --max-runtime on app submit, sql submit and connect server create, spec.max_runtime in job files, and a parameter of the SDK and both Airflow operators. Kubernetes stops the driver at the deadline, and diagnostics report deadline_exceeded. spark-on-k8s apps prune lists finished drivers and deletes them with --delete. #244
  • Assess Spark Operator SparkApplication and ScheduledSparkApplication manifests offline with spark-on-k8s job assess-operator. Each field is reported as mapped, approximated, unsupported or ignored, and --draft-dir writes draft job files. #254, #274
  • Record who asked the API for what: audit events, request IDs, and /metrics counters for audit events and Kubernetes errors. A gateway identity header can be recorded as an unverified claim with SPARK_ON_K8S_AUDIT_IDENTITY_HEADER. #253
  • Failed Airflow tasks log a bounded postmortem before cleanup, also pushed to XCom as spark_failure_diagnosis. #252
  • Serve the API below a gateway path with spark-on-k8s api start --root-path or SPARK_ON_K8S_API_ROOT_PATH. #233

Fixes

  • Submission and lifecycle: missing applications return a Kubernetes 404, waiters propagate Kubernetes errors instead of looping on stale status, explicit Kubernetes client configuration is used, async log polling is bounded and closes its watch, async kill clients close on errors and cancellation, and name-normalization warnings and error messages are accurate. Names without letters fall back to spark-app, list_apps skips drivers from other launchers, ConfigMap files are read as UTF-8, and a definitively rejected driver no longer leaves the Secret and ConfigMaps created for it. Kill and log streaming by application ID select only the driver Pod and its default container, and a final status read before a wait deadline is reported instead of a timeout. #147, #149, #151, #152, #153, #161, #164, #169, #185, #221, #226, #228
  • CLI: fractional executor allocation ratios are accepted, Kubernetes request timeouts exit with code 5, JSON output stays a single document for job-file and receipt errors, connect check uses the shared JSON contract, SQL files with a UTF-8 byte-order mark run, and bracketed IPv6 remote URLs use the right port. Uncertain remote submissions warn before a retry, Connect profiles reject a token without TLS (PySpark would switch to TLS), and missing api or Connect client extras produce an install hint. #158, #219, #226, #230
  • API and web UI: force=true is honored when deleting applications, an unconfigured history proxy returns HTTP 412, Kubernetes error statuses are preserved, and driver logs are rendered as literal text instead of HTML. Spark UI redirects stay on the public proxy, hop-by-hop headers are not forwarded, and console pages reference their assets by path, so a TLS gateway or spoofed Host header cannot break or redirect them. /metrics follows the Prometheus text format, an out-of-range history cursor returns 400, and the API image's entrypoint lets the server shut down gracefully. #154, #167, #170, #171, #202, #220, #229, #230
  • Airflow: executor allocation tuning is preserved, trigger errors are reported before deferred log replay, and uncertain adoption errors and partial submissions keep their identity. The SQL operator keeps user ConfigMaps, cleanup failures no longer replace the original task error, and the task tab distinguishes unavailable logs from an interrupted stream. #163, #186, #200, #222
  • Pre-release review, SDK and CLI: app submit without --image or --path and sql submit without --image are usage errors again; the async kill uses the manager's client configuration; the async log stream raises for a rejected log request instead of showing its error body as log text; a finished driver that never started ends log following with its status; remote plan|submit, job plan|submit and job validate report YAML-only, undecodable and unconstructible values as invalid input; text output closed by its reader ends quietly; python -m spark_on_k8s.cli works; spark-on-k8s and spark-on-k8s-dbt accept --version and -h; and app logs|kill|delete and namespace setup end with one error line instead of a traceback that could include the Kubernetes response body. #263, #278, #279
  • Pre-release review, Connect and diagnostics: an endpoint ending in ? or #, which made the client drop TLS, token and user, is rejected; the check runs with the caller's import paths, so user-site and PYTHONPATH installations work; connect server wait gives each check 15 seconds and reports the last failure on timeout; and cause_unknown names what a container is waiting for. #263, #272, #274
  • Pre-release review, API, console and run history: the Spark UI proxy closes the upstream when a header cannot be relayed (repeated failures exhausted its connection pool), answers 400 for targets it cannot forward and keeps repeated Set-Cookie headers; an unparseable spark-ui-proxy label no longer breaks the application list; the runs page opens a recorded namespace instead of an error; shutdown during lease acquisition releases the lease; log auto-scroll keeps following fast streams; a second Escape no longer hides a pending stop or delete; and slow History Server listings complete. Kubernetes refusing an application identifier or a continuation token is a 400 invalid_request, and an upload abandoned by its client is audited as rejected (client_disconnected) instead of an unknown submission. #263, #266, #276, #281
  • Pre-release review, Airflow: a failed log replay after deferral, a driver already deleted at cleanup, or a cleanup error after a synchronous failure no longer replaces the Spark outcome; an unknown app_waiter is rejected before submission; and the postmortem log tail redacts more secret spellings. A cleanup failure after Spark succeeded fails the task with a message naming Succeeded, so a retry is not mistaken for a Spark failure. #265, #274, #280
  • Pre-release review, chart: the documented quickstart values start the API under chart 0.4.0's default scope; names that read as YAML numbers or booleans stay strings; a values key left empty renders like its default; a kubeconfig selected in env is no longer overridden by in-cluster mode; and the helm test pod meets Pod Security restricted. #264, #271, #275
  • Pre-release review, dbt adapter: a failed Thrift session close no longer fails a successful model, and check --timeout nan is rejected. #267, #272
  • Security: offline plans redact executor environment values, cloud storage keys, SAS tokens, API keys and signed-URL parameters, and the History image downloads its build inputs over HTTPS only. #221, #229

Helm chart and other packages

  • Helm chart 0.4.0 will be published separately for this API version. It fixes in-cluster authentication and the listening address of the default deployment, namespace-scoped stop permissions, History Server rollouts and long-name collisions, and adds authenticated Helm tests. namespaces now defaults to the release namespace instead of "*": set namespaces: "*" before upgrading to keep the cluster-wide binding. The API pod runs as a non-root user with a Pod Security restricted context by default. The schema rejects --reuse-values upgrades from 0.3.0 that would drop new settings, the readiness probe allows the API's 5-second check, run-history namespaces must be covered by chart-managed RBAC, and the release workflow refuses an unpublished or pre-/ready API image. With namespaces: "*", the chart binds its own ClusterRole with the same rules as the namespaced Role instead of the built-in view or edit, and cluster-wide write requires rbac.allowClusterWideWrite: true. New optional values add a NetworkPolicy, PodDisruptionBudgets, a ServiceMonitor, the API worker count, History Server probes and remote submission from an existing Secret. CI rehearses the upgrade from chart 0.3.0 and the rollback. Thanks to @rcosperec for the original chart release work. #143, #157, #165, #218, #225, #232, #237, #247, #248, #249, #256, #260, #264, #271, #275
  • Experimental dbt adapter, dbt-spark-on-k8s 0.1.0. The separate distribution provisions temporary Spark compute for a dbt command over Thrift or Spark Connect and deletes it afterwards. It is versioned and released independently of this package, through a manual workflow that requires a released SDK. Interrupts, --fail-fast hooks, failed cancellations and lost tunnels are handled without leaking compute. spark-on-k8s-dbt check verifies a profile, the client libraries and the Kubernetes permissions without creating compute, and spark-on-k8s-dbt run can run several dbt commands on one managed server with --keep-going and --deadline. It is Experimental: any release, including a minor or patch one, may change its profile fields, commands or modules. #211, #212, #223, #227, #243, #259, #267

Documentation and maintenance

  • Reorganize the documentation around setup, running jobs, integrations, operations and reference. Each page shows its documentation channel and source commit, and this release-notes page is shared by every documentation version. Guides and examples were checked against the code, and the PyPI project description links to the published documentation. #184, #201, #217, #224, #233
  • Replace Poetry with uv and a committed lockfile. Test installed wheels for both Airflow versions, run Spark, Airflow, Connect, Jupyter, chart and dbt scenarios on disposable Kubernetes clusters, and qualify releases against the exact tagged commit. #142, #156, #178, #183, #187, #193, #200
  • Security maintenance: the Java example uses Spark 3.5.9, the dbt lockfile uses Thrift 0.25.0, and CodeQL alerts in a browser-test fixture are fixed. #193, #212, #213
  • New guides: when to use spark-on-k8s, migrating from the Spark Operator, ecosystem recipes (CronJob, batch schedulers, OpenLineage, S3 and MinIO), a single sign-on gateway with oauth2-proxy, an Airflow upgrade checklist for DAGs moving from 0.12, interface stability levels with 1.0 criteria, a security policy with private reporting, code owners, a maintainer ladder and an adopters list. #235, #236, #238, #240, #251, #254, #273
  • The published API image is the image that passed the release checks, signed by digest with keyless cosign in both registries. Dependabot proposes dependency updates, a report-only scan checks lockfiles and the API image for known vulnerabilities, and CI resolves the Airflow extras against Airflow's constraints files. #241, #257, #268, #269

Contributors

Thanks to @rcosperec for the Helm chart release work in #143, @parisni for the executor pod template implementation from #141 continued in #175, and @hussein-awala for the remaining implementation and maintenance work.

Full changelog: 0.12.1...0.13.0

Spark On K8S 0.12.1

Published 2025-04-07 · GitHub release

Bug Fixes:

Misc:

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.12.0...0.12.1

Spark On K8S 0.12.0

Published 2025-02-17 · GitHub release

Bug Fixes:

  • fix(airflow): adopt job only from the same dag_run_id/exection_date by @hussein-awala in #128

New Features:

  • feat(api): add 2 configs for delete and kill buttons in the webserver by @hussein-awala in #132
  • feat(api): add a config to control and disable refresh interval in apps list by @hussein-awala in #133

Misc:

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.11.0...0.12.0

Spark On K8S 0.11.0

Published 2025-01-19 · GitHub release

What's Changed

Significant Changes

Bug Fixes:

New Features:

  • feat(airflow): add on_finish_action to decide what to do when the job finishes by @hussein-awala in #100
  • feat(python-client): add more config for dynamic allocation by @hussein-awala in #102
  • feat(cli): add more config for dynamic allocation by @hussein-awala in #103
  • feat(airflow): create SparkSqlOnK8SOperator to execute Spark SQL commands by @hussein-awala in #109
  • feat(python-client): Support for init containers in driver pod by @hussein-awala in #121
  • feat(python-client): Support for host aliases in driver pod by @hussein-awala in #122

Misc

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.10.1...0.11.0

Spark On K8S 0.10.1

Published 2024-07-29 · GitHub release

What's Changed

  • fix(airflow): load namespace and pod_name from trigger event by @hussein-awala in #89

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.10.0...0.10.1

Spark On K8S 0.10.0

Published 2024-07-24 · GitHub release

What's Changed

  • security: bump zipp to avoid CVE-2024-5569 by @hussein-awala in #81
  • security: bump apache-airflow to avoid CVE-2024-39863 and CVE-2024-39877 by @hussein-awala in #82
  • security: update setuptools to avoid CVE-2024-6345 by @hussein-awala in #83
  • fix: propagate service account + test by @rivamarco in #88
  • feat(python-client, airflow): support startup_timeout for pending state waiting by @hussein-awala in #84

New Contributors

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.9.1...0.10.0

Spark On K8S 0.9.1

Published 2024-07-22 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.9.0...0.9.1

Spark On K8S 0.9.0

Published 2024-07-17 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.8.0...0.9.0

Spark On K8S 0.8.0

Published 2024-07-07 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.7.2...0.8.0

Spark On K8S 0.7.2

Published 2024-05-13 · GitHub release

What's Changed

  • fix(airflow): fix spark_history_ui_link by removing -driver suffix by @hussein-awala in #68

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.7.1...0.7.2

Spark On K8S 0.7.1

Published 2024-05-09 · GitHub release

What's Changed

  • fix(airflow): configure provider metadata to make the extra-link discoverable by @hussein-awala in #60
  • security: bump Jinja2 to avoid CVE-2024-34064 by @hussein-awala in #61

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.7.0...0.7.1

Spark On K8S 0.7.0

Published 2024-05-06 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.6.0...0.7.0

Spark On K8S 0.6.0

Published 2024-05-04 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.5.2...0.6.0

Spark On K8S 0.5.2

Published 2024-05-03 · GitHub release

What's Changed

  • fix(airflow): wait for termination status when there is no more log by @hussein-awala in #40

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.5.1...0.5.2

Spark On K8S 0.5.1

Published 2024-04-30 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.5.0...0.5.1

Spark On K8S 0.5.0

Published 2024-04-25 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.4.0...0.5.0

Spark On K8S 0.4.0

Published 2024-03-24 · GitHub release

What's Changed

This release introduces new features for the API and webserver, including the ability to kill and delete Spark jobs and read driver pod logs directly from the webserver, and some improvements for the helm chart access service account permissions.

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.3.0...0.4.0

Spark On K8S 0.3.0

Published 2024-03-04 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.2.3...0.3.0

Spark On K8S 0.2.3

Published 2024-02-28 · GitHub release

What's Changed

  • fix(cli): skip adding api commands when api extra is not installed by @hussein-awala in #8
  • fix(security): update flask-appbuilder to fix CVE-2024-25128 by @hussein-awala in #9

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.2.2...0.2.3

Spark On K8S 0.2.2

Published 2024-02-26 · GitHub release

What's Changed

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.2.1...0.2.2

Spark On K8S 0.2.1

Published 2024-02-24 · GitHub release

What's Changed

  • fix(security): bump cryptography version to fix CVE-2024-26130 by @hussein-awala in #2

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.2.0...0.2.1

Spark On K8S 0.2.0

Published 2024-02-18 · GitHub release

What's Changed

  • Support creating an ephemeral secret for sensitive arguments (Python client, CLI, and Airflow operator)
  • Support mounting existing secrets to the driver pod as environment variables (Python client and CLI)
  • Support mounting volumes to the spark driver pod (Python client and Airflow operator)
  • Support mounting volumes to the spark executor pods (Python client and Airflow operator)

Security

Full Changelog: https://github.com/hussein-awala/spark-on-k8s/compare/0.1.0...0.2.0