[Clang-Repl] Adds custom lambda in launchExecutor and PID retrieval

[kr-2003]

Continuation of #147478. Apologies for the confusion - I messed up with Git and needed to create a new PR to address

This PR introduces:

• Custom lambda function in launchExecutor.
• Fetching PID of the launched out-of-process(OOP) JIT executor.

[vgvassilev]

LGTM!

[rorth]

This patch broke the Solaris/amd64 bot.

[vgvassilev]

@kr-2003, we can probably disable the test for Solaris.

[rorth]

I don't think this is right: the test PASSes on Solaris/sparcv9, so this doesn't seem to be an OS issue.

[vgvassilev]

Is there any way to reproduce this?

[bartchr808]

This change is also breaking Bazel. After fixing the missing dependency in #152681, I'm now getting

Specified executor invalid: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/c9d34ded3a9d94cc250207948aceadfc/execroot/__main__/bazel-out/k8-fastbuild/bin/bin/llvm-jitlink-executor

which is raised in RemoteJITUtils.cpp.

https://buildkite.com/llvm-project/upstream-bazel/builds/145926/steps/canvas?sid=01988921-967b-46f5-91ce-2498c8480786

To reproduce, run bazel test --config=generic_clang @llvm-project//clang/unittests:interpreter_tests.

[kr-2003]

This change is also breaking Bazel. After fixing the missing dependency in #152681, I'm now getting

Specified executor invalid: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/c9d34ded3a9d94cc250207948aceadfc/execroot/__main__/bazel-out/k8-fastbuild/bin/bin/llvm-jitlink-executor

which is raised in RemoteJITUtils.cpp.

https://buildkite.com/llvm-project/upstream-bazel/builds/145926/steps/canvas?sid=01988921-967b-46f5-91ce-2498c8480786

To reproduce, run bazel test --config=generic_clang @llvm-project//clang/unittests:interpreter_tests.

I use this code for removing the extra dirs for calculating the executable paths:

llvm-project/clang/unittests/Interpreter/OutOfProcessInterpreterTests.cpp

Line 50 in 672f82a

static void removePathComponent(unsigned N, llvm::SmallString<256> &Path) {

here

llvm-project/clang/unittests/Interpreter/OutOfProcessInterpreterTests.cpp

Lines 55 to 60 in 672f82a

	static std::string getExecutorPath() {
	llvm::SmallString<256> ExecutorPath(llvm::sys::fs::getMainExecutable(
	nullptr, reinterpret_cast<void *>(&getExecutorPath)));
	removePathComponent(5, ExecutorPath);
	llvm::sys::path::append(ExecutorPath, "bin", "llvm-jitlink-executor");
	return ExecutorPath.str().str();

ALso, I can see 2 bins in the executor path.

..../bazel-out/k8-fastbuild/bin/bin/llvm-jitlink-executor

So, for bazel, I think the folder structure is quite different than expected.

[bartchr808]

It's likely adding the 2nd bin due to

llvm::sys::path::append(ExecutorPath, "bin", "llvm-jitlink-executor");

I suspect?

[bartchr808]

Ya changing that line to just

llvm::sys::path::append(ExecutorPath, "llvm-jitlink-executor");

I then get

Specified executor invalid: /{IGNORE}/.cache/bazel/_bazel_bartchr/16c24a85dd12eb0aa12cad671d68a98a/execroot/__main__/bazel-out/k8-fastbuild/bin/llvm-jitlink-executor

So still failing, but the expected path for bazel without the double /bin/bin/.

[kr-2003]

Ya changing that line to just

llvm::sys::path::append(ExecutorPath, "llvm-jitlink-executor");

I then get

Specified executor invalid: /{IGNORE}/.cache/bazel/_bazel_bartchr/16c24a85dd12eb0aa12cad671d68a98a/execroot/__main__/bazel-out/k8-fastbuild/bin/llvm-jitlink-executor

So still failing, but the expected path for bazel without the double /bin/bin/.

So, its seems that llvm-jitlink-executor is does not gets compiled for Bazel. Though, compiler-rt and HostSupportsJIT() conditions are satisfied.

llvm-project/clang/unittests/Interpreter/OutOfProcessInterpreterTests.cpp

Lines 133 to 146 in 3ea76af

	TEST_F(InterpreterTestBase, SanityWithRemoteExecution) {
	if (!HostSupportsJIT())
	GTEST_SKIP();
	std::unique_ptr<Interpreter> Interp = createInterpreterWithRemoteExecution();
	using PTU = PartialTranslationUnit;
	PTU &R1(cantFail(Interp->Parse("void g(); void g() {}")));
	EXPECT_EQ(2U, DeclsSize(R1.TUPart));
	PTU &R2(cantFail(Interp->Parse("int i;")));
	EXPECT_EQ(1U, DeclsSize(R2.TUPart));
	}

llvm-project/clang/unittests/Interpreter/CMakeLists.txt

Lines 9 to 14 in 3ea76af

	if(TARGET compiler-rt)
	list(APPEND CLANG_REPL_TEST_SOURCES
	OutOfProcessInterpreterTests.cpp
	)
	message(STATUS "Compiler-RT found, enabling out of process JIT tests")
	endif()

llvm-project/clang/unittests/Interpreter/CMakeLists.txt

Lines 41 to 47 in 3ea76af

	if(TARGET compiler-rt)
	add_dependencies(ClangReplInterpreterTests
	llvm-jitlink-executor
	compiler-rt
	)
	message(STATUS "Adding dependency on compiler-rt for out of process JIT tests")
	endif()

[kr-2003]

probably need to check for llvm-jitlink-executor target to be present

[bartchr808]

that file is part of the llvm-jitlink target here https://github.com/llvm/llvm-project/blob/main/utils/bazel/llvm-project-overlay/llvm/BUILD.bazel#L4206-L4207 if I add that on as a dependency to @llvm-project//clang/unittests:interpreter_tests, it's still failing with the same issue.

I'm not familiar really with what this PR is doing, could you take a look?

[bartchr808]

If you aren't familiar with bazel I can assist

[rupprecht]

Broadly speaking, bazel tests are much more explicit about paths, and you can make few assumptions about where things will be. A common fix is to have paths passed in as flags/env vars, or sometimes you can "copy" files (often just a symlink) so that it will be in the same dir. I think it's fine if we just skip this test for bazel for now.