Skip to content

Commit c8cb3bf

Browse files
committed
Handle perverse app names (with escaped quotes)
Now, the following works: bin/pyspark --master local --name "bat \" man \" is \" bat \" man"
1 parent 1866f85 commit c8cb3bf

File tree

1 file changed

+5
-6
lines changed

1 file changed

+5
-6
lines changed

bin/pyspark

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -63,16 +63,15 @@ if [ -n "$IPYTHON_OPTS" ]; then
6363
IPYTHON=1
6464
fi
6565

66-
# Build up arguments list manually to preserve quotes. We export Spark submit arguments as an
67-
# environment variable because shell.py must run as a PYTHONSTARTUP script, which does not take
68-
# in arguments. This is required mainly for IPython notebooks.
66+
# Build up arguments list manually to preserve quotes and backslashes.
67+
# We export Spark submit arguments as an environment variable because shell.py must run as a
68+
# PYTHONSTARTUP script, which does not take in arguments. This is required for IPython notebooks.
6969

7070
PYSPARK_SUBMIT_ARGS=""
7171
whitespace="[[:space:]]"
7272
for i in "$@"; do
73-
if [[ $i =~ $whitespace ]]; then
74-
i=\"$i\"
75-
fi
73+
if [[ $i =~ \" ]]; then i=$(echo $i | sed 's/\"/\\\"/g'); fi
74+
if [[ $i =~ $whitespace ]]; then i=\"$i\"; fi
7675
PYSPARK_SUBMIT_ARGS="$PYSPARK_SUBMIT_ARGS $i"
7776
done
7877
export PYSPARK_SUBMIT_ARGS

0 commit comments

Comments
 (0)