Celery queue related CI failures

[karajan1001]

Bug Report

Some of the celery queue-related tests failed, look into it I found this is because in some of the tests we regard the experiments to be finished after dvc.experiments.run(run_all=True)

dvc/tests/func/experiments/test_show.py

Lines 157 to 162 in 884182f

	dvc.experiments.run(
	failed_exp_stage.addressing, params=["foo=2"], queue=True
	)
	exp_rev = dvc.experiments.scm.resolve_rev(f"{CELERY_STASH}@{{0}}")
	dvc.experiments.run(run_all=True)
	experiments = dvc.experiments.show()[baseline_rev]

While this command will finish right after the sub-process is finished.

dvc/dvc/repo/experiments/queue/celery.py

Lines 321 to 323 in 884182f

	for line in self.proc.follow(entry.stash_rev, encoding):
	ui.write(line, end="")

And the task might still need some post process like (result collection and temp workspace cleaning)

dvc/dvc/repo/experiments/queue/tasks.py

Lines 108 to 111 in 884182f

	proc_dict = queue.proc.run_signature(cmd, name=entry.stash_rev)()
	collect_exp.s(proc_dict, entry_dict)()
	finally:
	cleanup_exp.s(executor, infofile)()

This will probably let the tasks still in running status in the following commands.

Description

Reproduce

Expected

Environment information

Output of dvc doctor:

$ dvc doctor

Additional Information (if any):