Skip to content

[SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext #52402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext #52402

wants to merge 2 commits into from
+150 −87

Conversation

ganeshashree
Copy link

What changes were proposed in this pull request?

Refactor MemoryStream to use SparkSession instead of SQLContext.

Why are the changes needed?

SQLContext is deprecated in newer versions of Spark.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Verified that the affected tests are passing successfully.

Was this patch authored or co-authored using generative AI tooling?

No

cloud-fan
cloud-fan reviewed Sep 22, 2025
View reviewed changes
...lines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala

test("three hop pipeline") {
val session = spark
implicit val sparkSession: SparkSession = spark
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where was the previous implicit SQLContext defined?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like it was getting implicit sqlContext defined in SharedSparkSession.

@cloud-fan
Copy link
Contributor

cc @HeartSaVioR

@HeartSaVioR
Copy link
Contributor

@ganeshashree
Thanks for the proposal. The change looks OK to me.

Have we checked the warn (build/log) message when we use SQLContext here? If we weren't providing the message to migrate easily, it might be beneficial to defer replacement of apply() and have intermediate migration step (deprecation of the existing methods and removal of them in Spark 5.0.0).

@HeartSaVioR HeartSaVioR changed the title [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext Sep 22, 2025
@ganeshashree
Copy link
Author

ganeshashree commented Sep 29, 2025 •
edited
Loading

@ganeshashree Thanks for the proposal. The change looks OK to me.

Have we checked the warn (build/log) message when we use SQLContext here? If we weren't providing the message to migrate easily, it might be beneficial to defer replacement of apply() and have intermediate migration step (deprecation of the existing methods and removal of them in Spark 5.0.0).

@HeartSaVioR Thanks for reviewing. Currently, no warning appears in the build log when we use SQLContext. Creating two versions of MemoryStream.apply for SparkSession and SQLContext and showing a warning for SQLContext would require resolving ambiguity when both sparkSession and sqlContext are set as implicit variables. Since this is an internal API, please review whether it's acceptable to make this change and update the callers to use MemoryStream with an implicit SparkSession instead of SQLContext, where applicable. I'm exploring further to resolve the ambiguity by preferring SparkSession instead of SQLContext.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

Assignees
No one assigned
Labels
SQL STRUCTURED STREAMING
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ganeshashree @cloud-fan @HeartSaVioR @ganeshas-db