operator resubmits all spark applications after restart no matter what their status is

[maxgruber19]

Affected Stackable version

24.7

Affected Apache Spark-on-Kubernetes version

3.4.2

Current and expected behavior

when the spark-k8s operator restarts (e.g. in case of node restart or operator upgrade) every spark application is reconciled/resubmitted at once. this leads to hell and chaos because of both resource consumption and multiple jobs writing / reading one file which usually done ordered by airflow dags doing the spark submits

Possible solution

not submitting an application which is in state succeeded / failed / stopped / killed. maybe even running apps should not be restarted because they are already running

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[maxgruber19]

@razvan solution sounds good, thanks for having a look at it

[lfrancke]

@razvan could you please add a short snippet we can use for the release notes?

[razvan]

@razvan could you please add a short snippet we can use for the release notes?

Spark Operator Bugfix: ensure Spark applications are submitted only once. Reconciling applications after the corresponding Job objects have been recycled doesn't lead to the creation of new Job objects. This behavior was triggered by different situations, such as when the operator was restarted.