Skip to content

Conversation

sbernauer
Copy link
Member

@sbernauer sbernauer commented May 26, 2025

This also uncovered two bugs:

  1. spark-connect-controller does not watch StatefulSets here
  2. The label selectors app.kubernetes.io/name=spark-connect,app.kubernetes.io/instance=spark-connect und app.kubernetes.io/name=spark-history,app.kubernetes.io/instance=spark-history match nothing. We should set app.kubernetes.io/name=spark-connect instead of currently app.kubernetes.io/name: spark-k8s (same for HistoryServer)

I pinged @razvan about the bugs, once they are fixed everything should work
=> He created stackabletech/spark-k8s-operator#573

┌───────────────┬───────────────┬───────────┬──────────────────────────────────────────────────────────────┬─────────────────────────────────┐
│ PRODUCT       ┆ NAME          ┆ NAMESPACE ┆ ENDPOINTS                                                    ┆ CONDITIONS                      │
╞═══════════════╪═══════════════╪═══════════╪══════════════════════════════════════════════════════════════╪═════════════════════════════════╡
│ hdfs          ┆ hdfs          ┆ default   ┆ datanode-default-0-listener-data     172.18.0.2:31278        ┆ Available, Reconciling, Running │
│               ┆               ┆           ┆ datanode-default-0-listener-http     http://172.18.0.2:32716 ┆                                 │
│               ┆               ┆           ┆ datanode-default-0-listener-ipc      172.18.0.2:32057        ┆                                 │
│               ┆               ┆           ┆ datanode-default-0-listener-metrics  172.18.0.2:31053        ┆                                 │
│               ┆               ┆           ┆ namenode-default-0-http              http://172.18.0.2:31589 ┆                                 │
│               ┆               ┆           ┆ namenode-default-0-metrics           172.18.0.2:31089        ┆                                 │
│               ┆               ┆           ┆ namenode-default-0-rpc               172.18.0.2:31233        ┆                                 │
│               ┆               ┆           ┆ namenode-default-1-http              http://172.18.0.2:31372 ┆                                 │
│               ┆               ┆           ┆ namenode-default-1-metrics           172.18.0.2:31917        ┆                                 │
│               ┆               ┆           ┆ namenode-default-1-rpc               172.18.0.2:31921        ┆                                 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ spark-connect ┆ spark-connect ┆ default   ┆                                                              ┆ Available, Reconciling, Running │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ zookeeper     ┆ zookeeper     ┆ default   ┆                                                              ┆ Available, Reconciling, Running │
└───────────────┴───────────────┴───────────┴──────────────────────────────────────────────────────────────┴─────────────────────────────────┘

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes
# Author
- [ ] Changes are OpenShift compatible
- [ ] CRD changes approved
- [ ] Helm chart can be installed and deployed operator works
- [ ] Integration tests passed (for non trivial changes)
# Reviewer
- [ ] Code contains useful comments
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)
# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added

@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering May 26, 2025
@maltesander maltesander self-requested a review May 27, 2025 08:10
@maltesander maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering May 27, 2025
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sbernauer sbernauer added this pull request to the merge queue May 27, 2025
@sbernauer sbernauer moved this from Development: In Review to Development: Done in Stackable Engineering May 27, 2025
Merged via the queue into main with commit 86248de May 27, 2025
21 of 22 checks passed
@sbernauer sbernauer deleted the chore/support-spark-connect branch May 27, 2025 10:46
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering May 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants