[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic #46155

HyukjinKwon · 2024-04-22T05:11:33Z

What changes were proposed in this pull request?

Same as #46129 but for Column class.

Why are the changes needed?

Same as #46129

Does this PR introduce any user-facing change?

Same as #46129

How was this patch tested?

Manually tested, and CI should verify them.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun

What is the meaning of pyspark.sql.classic?

HyukjinKwon · 2024-04-22T09:34:25Z

Let me run this naming stuff separately (maybe sending an email to the dev mailing list) because Spark Classic has been used in other places lately in any event.

dongjoon-hyun · 2024-04-22T17:12:38Z

Sure, please proceed it. We can discuss the naming separately, @HyukjinKwon .

ueshin · 2024-04-22T18:44:38Z

python/pyspark/ml/connect/functions.py

@@ -15,7 +15,7 @@
 # limitations under the License.
 #
 from pyspark.ml import functions as PyMLFunctions
-from pyspark.sql.connect.column import Column
+from pyspark.sql.column import Column


Should keep it as-is to be consistent with the classic functions.py if it changes to refer classic?

It is actually related to the type hints. We're using pyspark.sql.Column for type hints, and when we actually need to create the instance, we should use pyspark.sql.classic.column import Column or pyspark.sql.connect.column import Column .. it is also a bit messy. I should probably clean those up separately.

ueshin · 2024-04-22T18:49:39Z

python/pyspark/ml/stat.py

@@ -22,7 +22,8 @@
 from pyspark.ml.common import _java2py, _py2java
 from pyspark.ml.linalg import Matrix, Vector
 from pyspark.ml.wrapper import JavaWrapper, _jvm
-from pyspark.sql.column import Column, _to_seq
+from pyspark.sql.column import Column


Is this consistent with functions.py?

or import _to_seq later as the other files do?

This file is only used in Spark Classic so it should probably be fine. It is a bit messy and complicated .. I will try to refactor them separately.

HyukjinKwon · 2024-04-23T05:12:44Z

Merged to master.

For naming, I will send an email soon.
For others, please leave the comments. I will followup.

…t `_with_origin` ### What changes were proposed in this pull request? This PR is a followup of #46155 that adds check of `__name__` at `_with_origin`. ### Why are the changes needed? It is possible for a callable instance without __name__ attribute or/and __module__ attribute to be wrapped. For example, functools.partial. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `./bin/pyspark` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46198 from HyukjinKwon/SPARK-47933-followup2. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…n `pyspark-connect` ### What changes were proposed in this pull request? This PR is a followup of #46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8821919392/job/24218893631 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46229 from HyukjinKwon/SPARK-47933-followuptmp. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…` reference in `pyspark.ml.stat` ### What changes were proposed in this pull request? This PR is a followup of #46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8861971303 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46262 from HyukjinKwon/SPARK-47933-followup4. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

… classes ### What changes were proposed in this pull request? This PR proposes to fix `pyspark-connect` build that is broken by #46155. It is similar with the followups such as #46229 and #46262 but make sure the tests pass within this PR. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8884003967 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released yet. ### How was this patch tested? CI in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8889698252/job/24408487227 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46294 from HyukjinKwon/SPARK-48052. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

… Classic ### What changes were proposed in this pull request? Same as apache#46129 but for `Column` class. ### Why are the changes needed? Same as apache#46129 ### Does this PR introduce _any_ user-facing change? Same as apache#46129 ### How was this patch tested? Manually tested, and CI should verify them. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46155 from HyukjinKwon/SPARK-47933. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…t `_with_origin` ### What changes were proposed in this pull request? This PR is a followup of apache#46155 that adds check of `__name__` at `_with_origin`. ### Why are the changes needed? It is possible for a callable instance without __name__ attribute or/and __module__ attribute to be wrapped. For example, functools.partial. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `./bin/pyspark` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46198 from HyukjinKwon/SPARK-47933-followup2. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…n `pyspark-connect` ### What changes were proposed in this pull request? This PR is a followup of apache#46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8821919392/job/24218893631 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46229 from HyukjinKwon/SPARK-47933-followuptmp. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…` reference in `pyspark.ml.stat` ### What changes were proposed in this pull request? This PR is a followup of apache#46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8861971303 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46262 from HyukjinKwon/SPARK-47933-followup4. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

… classes ### What changes were proposed in this pull request? This PR proposes to fix `pyspark-connect` build that is broken by apache#46155. It is similar with the followups such as apache#46229 and apache#46262 but make sure the tests pass within this PR. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8884003967 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released yet. ### How was this patch tested? CI in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8889698252/job/24408487227 ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46294 from HyukjinKwon/SPARK-48052. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

github-actions bot added SQL ML STRUCTURED STREAMING PYTHON AVRO PANDAS API ON SPARK CONNECT PROTOBUF labels Apr 22, 2024

HyukjinKwon force-pushed the SPARK-47933 branch from ae50ce3 to 0bd565b Compare April 22, 2024 05:29

github-actions bot added the BUILD label Apr 22, 2024

HyukjinKwon force-pushed the SPARK-47933 branch 6 times, most recently from 5fa46be to a6677c5 Compare April 22, 2024 06:28

zhengruifeng approved these changes Apr 22, 2024

View reviewed changes

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

HyukjinKwon added 3 commits April 22, 2024 18:33

Parent Column class for Spark Connect and Spark Classic

9576489

fixup

0f04479

fixup

3aebec4

fixup

7dde1e8

HyukjinKwon force-pushed the SPARK-47933 branch from beddbbf to 7dde1e8 Compare April 22, 2024 09:37

HyukjinKwon added 2 commits April 22, 2024 19:19

fixup

dadf9c7

fixup

4707269

ueshin reviewed Apr 22, 2024

View reviewed changes

HyukjinKwon added 2 commits April 23, 2024 07:51

fixup

fe4368e

fixup

91d4bd6

HyukjinKwon closed this in e01ac58 Apr 23, 2024

HyukjinKwon mentioned this pull request Apr 24, 2024

[SPARK-47933][PYTHON][CONNECT][FOLLOW-UP] Add a check of __name__ at _with_origin #46198

Closed

HyukjinKwon mentioned this pull request Apr 25, 2024

[SPARK-47933][CONNECT][PYTHON][FOLLOW-UP] Avoid referencing _to_seq in pyspark-connect #46229

Closed

HyukjinKwon mentioned this pull request Apr 28, 2024

[SPARK-47933][CONNECT][PYTHON][FOLLOW-UP] Remove pyspark.sql.classic reference in pyspark.ml.stat #46262

Closed

HyukjinKwon mentioned this pull request Apr 30, 2024

[SPARK-48052][PYTHON][CONNECT] Recover pyspark-connect CI by parent classes #46294

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic #46155

[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic #46155

Uh oh!

HyukjinKwon commented Apr 22, 2024

Uh oh!

dongjoon-hyun left a comment •

edited

Loading

Uh oh!

HyukjinKwon commented Apr 22, 2024

Uh oh!

dongjoon-hyun commented Apr 22, 2024

Uh oh!

ueshin Apr 22, 2024

Uh oh!

HyukjinKwon Apr 23, 2024

Uh oh!

ueshin Apr 22, 2024

Uh oh!

ueshin Apr 22, 2024

Uh oh!

HyukjinKwon Apr 23, 2024

Uh oh!

HyukjinKwon commented Apr 23, 2024

Uh oh!

Uh oh!

[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic #46155

[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic #46155

Uh oh!

Conversation

HyukjinKwon commented Apr 22, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Apr 22, 2024

Uh oh!

dongjoon-hyun commented Apr 22, 2024

Uh oh!

ueshin Apr 22, 2024

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Apr 23, 2024

Choose a reason for hiding this comment

Uh oh!

ueshin Apr 22, 2024

Choose a reason for hiding this comment

Uh oh!

ueshin Apr 22, 2024

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Apr 23, 2024

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Apr 23, 2024

Uh oh!

Uh oh!

dongjoon-hyun left a comment •

edited

Loading